Oct 31 14:45:15.252977 ip-10-0-137-160 systemd[1]: Starting Kubernetes Kubelet... Oct 31 14:45:15.650737 ip-10-0-137-160 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:15.650737 ip-10-0-137-160 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 14:45:15.650737 ip-10-0-137-160 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:15.650737 ip-10-0-137-160 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:15.656068 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.655932 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:15.656068 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656066 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:15.656068 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656069 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:15.656068 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656072 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:15.656068 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656075 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656078 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656081 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656084 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656086 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656089 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656092 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656095 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656097 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656100 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656103 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656106 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656108 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656111 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656114 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656118 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656122 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656125 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656128 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:15.656261 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656131 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656133 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656136 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656139 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656142 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656144 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656148 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656150 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656153 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656155 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656158 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656161 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656163 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656166 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656169 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656172 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656175 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656177 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656180 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656182 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:15.656763 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656185 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656187 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656190 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656193 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656195 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656197 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656200 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656202 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656205 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656207 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656209 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656212 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656215 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656218 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656220 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656223 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656225 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656228 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656231 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656233 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:15.657227 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656235 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656238 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656241 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656244 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656246 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656249 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656251 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656254 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656256 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656259 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656261 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656264 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656267 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656269 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656272 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656274 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656277 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656280 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656282 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:15.657681 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656285 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656288 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656292 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656296 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656299 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656302 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656304 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656308 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656310 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656313 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656315 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656318 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656320 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656323 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656325 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656328 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656331 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656333 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656336 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:15.658137 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656338 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656344 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656347 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656349 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.656352 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657142 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657150 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657153 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657156 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657158 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657161 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657164 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657167 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657169 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657172 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657180 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657183 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657186 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657189 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:15.658572 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657192 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657194 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657197 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657199 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657202 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657204 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657207 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657210 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657212 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657215 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657218 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657220 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657223 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657225 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657229 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657231 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657234 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657237 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657239 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657242 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:15.659073 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657244 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657247 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657249 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657252 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657254 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657258 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657262 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657264 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657267 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657270 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657273 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657276 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657279 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657282 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657284 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657287 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657290 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657292 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657294 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657298 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:15.659543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657300 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657303 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657305 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657308 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657310 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657313 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657315 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657318 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657320 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657323 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657325 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657328 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657331 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657333 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657336 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657338 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657341 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657343 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657346 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657348 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:15.660025 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657351 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657353 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657355 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657358 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657360 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657363 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657365 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657368 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657370 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657372 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657375 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657378 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657380 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657383 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657385 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657388 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657390 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657393 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657395 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:15.660494 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657398 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657400 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657405 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657409 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657412 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657415 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657417 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657419 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657422 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657425 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657427 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657430 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657432 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657924 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657930 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657934 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657937 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657939 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657942 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:15.660960 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657945 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657947 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657950 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657952 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657955 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657958 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657960 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657963 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657966 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657968 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657971 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657973 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657976 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657979 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657982 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657984 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657987 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657989 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657992 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657995 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:15.661389 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.657997 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658000 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658002 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658005 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658007 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658009 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658012 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658014 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658017 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658023 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658026 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658029 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658031 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658034 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658038 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658042 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658046 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658050 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658053 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:15.661857 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658056 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658059 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658061 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658064 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658066 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658069 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658071 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658074 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658076 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658079 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658081 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658083 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658086 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658089 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658091 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658094 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658097 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658099 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658101 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658104 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:15.662305 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658106 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658108 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658111 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658119 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658122 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658126 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658129 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658131 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658133 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658137 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658140 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658143 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658145 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658148 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658150 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658153 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658155 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658158 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658160 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:15.662804 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658164 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658167 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658170 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658172 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658175 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658177 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658180 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658182 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658185 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658187 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658190 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658192 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658194 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658197 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658199 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658202 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658204 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658207 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658211 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:15.663242 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658213 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658215 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.658218 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659255 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659266 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659273 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659278 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659282 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659286 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659290 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659294 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659298 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659300 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659304 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659307 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659310 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659313 2578 flags.go:64] FLAG: --cgroup-root="" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659316 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659319 2578 flags.go:64] FLAG: --client-ca-file="" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659322 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659325 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659329 2578 flags.go:64] FLAG: --cluster-domain="" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659331 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 14:45:15.663680 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659334 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659337 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659341 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659345 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659348 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659351 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659354 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659357 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659361 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659364 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659367 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659370 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659376 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659379 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659381 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659384 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659387 2578 flags.go:64] FLAG: --enable-server="true" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659390 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659394 2578 flags.go:64] FLAG: --event-burst="100" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659398 2578 flags.go:64] FLAG: --event-qps="50" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659401 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659404 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659406 2578 flags.go:64] FLAG: --eviction-hard="" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659411 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 14:45:15.664226 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659414 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659417 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659420 2578 flags.go:64] FLAG: --eviction-soft="" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659423 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659426 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659428 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659431 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659434 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659437 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659440 2578 flags.go:64] FLAG: --feature-gates="" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659444 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659448 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659451 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659454 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659457 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659460 2578 flags.go:64] FLAG: --help="false" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659463 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659467 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659470 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659473 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659477 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659480 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659483 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 14:45:15.664862 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659486 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659489 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659492 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659495 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659498 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659501 2578 flags.go:64] FLAG: --kube-reserved="" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659504 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659506 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659509 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659512 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659515 2578 flags.go:64] FLAG: --lock-file="" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659517 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659520 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659523 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659529 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659531 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659535 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659538 2578 flags.go:64] FLAG: --logging-format="text" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659540 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659544 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659547 2578 flags.go:64] FLAG: --manifest-url="" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659551 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659555 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659558 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659562 2578 flags.go:64] FLAG: --max-pods="110" Oct 31 14:45:15.665397 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659564 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659567 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659570 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659573 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659577 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659580 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659583 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659590 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659593 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659596 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659599 2578 flags.go:64] FLAG: --pod-cidr="" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659602 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659604 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659608 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659610 2578 flags.go:64] FLAG: --port="10250" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659613 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659616 2578 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b98ce92b757cca08" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659620 2578 flags.go:64] FLAG: --qos-reserved="" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659623 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659626 2578 flags.go:64] FLAG: --register-node="true" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659628 2578 flags.go:64] FLAG: --register-with-taints="" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659632 2578 flags.go:64] FLAG: --registry-burst="10" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659636 2578 flags.go:64] FLAG: --registry-qps="5" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659639 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659642 2578 flags.go:64] FLAG: --reserved-memory="" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659646 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 14:45:15.666115 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659649 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659652 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659655 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659658 2578 flags.go:64] FLAG: --runonce="false" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659661 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659664 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659667 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659670 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659672 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659676 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659680 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659683 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659686 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659689 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659692 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659708 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659711 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659714 2578 flags.go:64] FLAG: --system-cgroups="" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659717 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659722 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659726 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659728 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659732 2578 flags.go:64] FLAG: --tls-min-version="" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659735 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659738 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 14:45:15.666791 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659740 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659743 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659746 2578 flags.go:64] FLAG: --v="2" Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659752 2578 flags.go:64] FLAG: --version="false" Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659756 2578 flags.go:64] FLAG: --vmodule="" Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659760 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.659763 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659893 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659897 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659899 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659902 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659904 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659907 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659910 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659912 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659915 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659917 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659922 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659924 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659927 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659929 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:15.667380 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659932 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659934 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659937 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659939 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659942 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659944 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659947 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659950 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659952 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659954 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659957 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659959 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659962 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659964 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659968 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659970 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659973 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659975 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659978 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:15.667918 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659980 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659982 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659985 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659987 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659990 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659992 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659995 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659997 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.659999 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660003 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660006 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660008 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660011 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660013 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660016 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660019 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660021 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660024 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660026 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:15.668358 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660029 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660031 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660034 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660037 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660039 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660042 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660044 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660047 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660052 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660055 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660057 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660060 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660062 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660065 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660067 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660070 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660072 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660074 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660077 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660080 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:15.668921 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660082 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660085 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660088 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660091 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660093 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660096 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660100 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660104 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660108 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660111 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660114 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660116 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660119 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660121 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660124 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660126 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660128 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660131 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660134 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:15.669384 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660136 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660140 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660143 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660145 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660148 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660150 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660153 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660155 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660158 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660160 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660163 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660165 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660167 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660170 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660172 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.660199 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:15.669841 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660235 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660240 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660244 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660247 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660250 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660252 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660255 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660258 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660260 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660263 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660265 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660268 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660270 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660273 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660275 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660278 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660280 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660283 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660286 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660289 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:15.670213 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660291 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660294 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660296 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660299 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660302 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660305 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660307 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660309 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660312 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660314 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660318 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660328 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660331 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660335 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660338 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660340 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660344 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660347 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660349 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:15.670688 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660351 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660354 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660357 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660359 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660362 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660364 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660367 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660369 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660372 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660374 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660376 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660380 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660383 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660385 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660388 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660390 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660393 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660395 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660398 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660400 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:15.671139 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660403 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660405 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660408 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660410 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660413 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660416 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660418 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660421 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660423 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660426 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660429 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660431 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660433 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660436 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660438 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660441 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660443 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660446 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660448 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:15.671599 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660450 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660453 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660456 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660458 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660462 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660464 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660467 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660469 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660472 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660474 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660477 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660479 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660482 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660484 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660486 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660489 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660491 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660496 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660499 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:15.672051 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660502 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660504 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660507 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660509 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660515 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660518 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660520 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660523 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.660526 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.667846 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.667859 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.668611 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.668617 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668671 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668677 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668680 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668683 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668685 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668688 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:15.672545 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668691 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668712 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668716 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668720 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668725 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668728 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668731 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668734 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668737 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668739 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668742 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668744 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668747 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668750 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668752 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668755 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668758 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668761 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668764 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:15.672997 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668767 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668769 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668772 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668775 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668778 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668780 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668783 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668786 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668788 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668791 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668793 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668796 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668798 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668801 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668803 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668806 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668808 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668811 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668813 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:15.673527 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668815 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668818 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668820 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668823 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668825 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668828 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668830 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668832 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668835 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668837 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668839 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668842 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668845 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668848 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668850 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668852 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668855 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668858 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668861 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668863 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:15.673992 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668866 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668869 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668873 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668876 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668879 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668881 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668883 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668886 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668888 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668891 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668893 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668896 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668898 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668900 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668903 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668905 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668908 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668910 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668913 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668915 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:15.674537 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668917 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668920 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668923 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668925 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668928 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668931 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668934 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668937 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668939 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668942 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668945 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668947 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668949 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668952 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668954 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668956 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668959 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668961 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668963 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668966 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:15.675055 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668968 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.668971 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.668998 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669135 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669139 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669142 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669145 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669148 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669151 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669153 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669156 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669159 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669161 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669163 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669166 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669168 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:15.675513 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669171 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669173 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669176 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669179 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669182 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669184 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669186 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669189 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669191 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669194 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669196 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669198 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669201 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669203 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669205 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669208 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669210 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669212 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669215 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:15.676031 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669217 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669220 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669222 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669224 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669227 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669229 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669232 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669234 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669236 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669239 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669241 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669244 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669246 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669248 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669251 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669253 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669256 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669259 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669261 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:15.676806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669264 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669267 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669272 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669275 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669278 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669280 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669283 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669285 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669287 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669290 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669292 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669295 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669297 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669299 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669302 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669304 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669307 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669309 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669312 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:15.677313 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669315 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669317 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669320 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669322 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669324 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669327 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669329 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669332 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669334 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669336 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669339 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669342 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669344 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669347 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669350 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669354 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669357 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669359 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669362 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:15.677817 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669364 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669367 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669370 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669372 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669375 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669377 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669380 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669382 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669385 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669388 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669391 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669394 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669396 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669399 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669401 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669403 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:15.678436 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669406 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.669429 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669461 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669465 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669469 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669472 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669474 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669477 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669479 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669482 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669485 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669487 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669490 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669492 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669495 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669497 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:15.678990 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669499 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669502 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669504 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669507 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669509 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669511 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669515 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669520 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669523 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669525 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669528 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669531 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669533 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669535 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669538 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669540 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669543 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669545 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669547 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:15.679470 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669550 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669552 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669555 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669558 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669560 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669562 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669565 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669568 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669571 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669573 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669575 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669578 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669580 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669583 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669585 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669587 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669590 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669592 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669594 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:15.679995 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669598 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669601 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669604 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669606 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669609 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669612 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669614 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669617 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669619 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669622 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669625 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669627 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669629 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669632 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669635 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669637 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669640 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669643 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669645 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:15.680578 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669647 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669650 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669652 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669655 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669657 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669660 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669662 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669665 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669667 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669670 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669672 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669674 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669677 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669679 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669681 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669684 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669686 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669691 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669710 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669715 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:15.681022 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669718 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669721 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669723 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669726 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669728 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669730 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669735 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669737 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669740 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669742 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669745 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669747 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669750 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669752 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:15.669755 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.669802 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.672187 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.673223 2578 server.go:1014] "Starting client certificate rotation" Oct 31 14:45:15.681602 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.673348 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 14:45:15.682095 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.674053 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 14:45:15.682095 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.675254 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 14:45:15.682095 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.676402 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 14:45:15.687506 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.687491 2578 log.go:25] "Validated CRI v1 runtime API" Oct 31 14:45:15.691334 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.691320 2578 log.go:25] "Validated CRI v1 image API" Oct 31 14:45:15.692176 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.692148 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 14:45:15.696095 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.696077 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d8c27f2b-ce49-4d2b-9965-a422c97d2075:/dev/nvme0n1p3 e2843bd1-6e4b-45bc-aad3-42c48956d871:/dev/nvme0n1p4] Oct 31 14:45:15.696154 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.696095 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 14:45:15.702756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.702639 2578 manager.go:217] Machine: {Timestamp:2026-10-31 14:45:15.701156544 +0000 UTC m=+0.340674766 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101426 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25cc9045f0cd0e61b04ba0bc4de700 SystemUUID:ec25cc90-45f0-cd0e-61b0-4ba0bc4de700 BootID:aee4a6c9-0428-4584-aea0-dc95f9cb6a86 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:4f:44:c6:8d:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:4f:44:c6:8d:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:06:27:51:23:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 14:45:15.702756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.702758 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 14:45:15.702859 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.702835 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 14:45:15.706317 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.706284 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 14:45:15.706444 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.706320 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-160.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 14:45:15.706493 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.706446 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 14:45:15.706493 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.706453 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 14:45:15.706545 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.706496 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 14:45:15.706545 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.706514 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 14:45:15.707676 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.707666 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 14:45:15.712197 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.712185 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 14:45:15.712320 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.712311 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 14:45:15.713126 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.713115 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 31 14:45:15.713170 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.713128 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 14:45:15.713170 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.713151 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 14:45:15.713170 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.713160 2578 kubelet.go:405] "Adding apiserver pod source" Oct 31 14:45:15.713291 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.713177 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 14:45:15.713510 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.713499 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 14:45:15.719009 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.718978 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 14:45:15.726243 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.726230 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 14:45:15.727146 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727135 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 14:45:15.727188 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727164 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 14:45:15.727328 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727310 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 14:45:15.727404 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727331 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 14:45:15.727404 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727340 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 14:45:15.727404 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727347 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 14:45:15.727404 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727356 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 14:45:15.727404 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727363 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 14:45:15.727990 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727981 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 14:45:15.728019 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727991 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 14:45:15.728019 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.727998 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 14:45:15.728019 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.728004 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 14:45:15.728019 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.728012 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 14:45:15.728173 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.728021 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 14:45:15.729459 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.729450 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 14:45:15.729490 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.729460 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 14:45:15.733662 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.733646 2578 server.go:1263] "Started kubelet" Oct 31 14:45:15.733784 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.733756 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 14:45:15.733784 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.733750 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 14:45:15.733856 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.733810 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 14:45:15.734465 ip-10-0-137-160 systemd[1]: Started Kubernetes Kubelet. Oct 31 14:45:15.735325 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.735310 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 14:45:15.740995 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.740976 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 31 14:45:15.742279 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:15.742227 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 14:45:15.746541 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.746516 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 14:45:15.747108 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.747081 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 14:45:15.747193 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:15.747170 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:15.747908 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.747887 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 14:45:15.748061 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748046 2578 factory.go:55] Registering systemd factory Oct 31 14:45:15.748106 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748067 2578 factory.go:223] Registration of the systemd container factory successfully Oct 31 14:45:15.748156 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748127 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 14:45:15.748156 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748139 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 14:45:15.748253 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748240 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 31 14:45:15.748300 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748254 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 31 14:45:15.748344 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748319 2578 factory.go:153] Registering CRI-O factory Oct 31 14:45:15.748344 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748334 2578 factory.go:223] Registration of the crio container factory successfully Oct 31 14:45:15.748434 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748384 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 14:45:15.748434 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748402 2578 factory.go:103] Registering Raw factory Oct 31 14:45:15.748434 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.748418 2578 manager.go:1199] Started watching for new ooms in manager Oct 31 14:45:15.749629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.749555 2578 manager.go:319] Starting recovery of all containers Oct 31 14:45:15.759645 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.759630 2578 manager.go:324] Recovery completed Oct 31 14:45:15.763239 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.763224 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:15.765788 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.765767 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:15.765864 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.765807 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:15.765864 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.765823 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:15.766374 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.766361 2578 cpu_manager.go:226] "Starting" policy="none" Oct 31 14:45:15.766434 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.766374 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 14:45:15.766434 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.766364 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:15.766434 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.766395 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 14:45:15.767072 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.767057 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:15.767165 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.767090 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:15.767165 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.767104 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:15.768595 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.768582 2578 policy_none.go:50] "Start" Oct 31 14:45:15.768644 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.768599 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 14:45:15.768644 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.768618 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 14:45:15.770550 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.770538 2578 policy_none.go:44] "Start" Oct 31 14:45:15.816926 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.816790 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 14:45:15.829023 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.829006 2578 manager.go:341] "Starting Device Plugin manager" Oct 31 14:45:15.829111 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:15.829036 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 14:45:15.829111 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.829053 2578 server.go:87] "Starting device plugin registration server" Oct 31 14:45:15.829316 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.829303 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 14:45:15.829361 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.829317 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 14:45:15.829504 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.829487 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 14:45:15.829584 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.829572 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 14:45:15.829649 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.829587 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 14:45:15.830047 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:15.830027 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 14:45:15.830115 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:15.830083 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:15.874754 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.874729 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-160.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 14:45:15.876760 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:15.874727 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-160.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 14:45:15.876760 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:15.874732 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-160.us-west-1.compute.internal.18e3a478e2ea2a96 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-160.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-160.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 14:45:15.733617302 +0000 UTC m=+0.373135524,LastTimestamp:2026-10-31 14:45:15.733617302 +0000 UTC m=+0.373135524,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-160.us-west-1.compute.internal,}" Oct 31 14:45:15.880741 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.880717 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 14:45:15.882014 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.882001 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 14:45:15.882064 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.882020 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 14:45:15.882064 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.882036 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 14:45:15.882134 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:15.882068 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 14:45:15.939722 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.939638 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:15.940593 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.940576 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:15.940660 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.940612 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:15.940660 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.940630 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:15.940660 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.940657 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:15.982561 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.982542 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal"] Oct 31 14:45:15.982635 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.982595 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:15.983381 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.983365 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:15.983469 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.983400 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:15.983469 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.983415 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:15.985781 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.985767 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:15.986330 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.986308 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:15.986422 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.986341 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:15.986509 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.986428 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:15.986509 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.986466 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:15.986509 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.986478 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:15.987020 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.987006 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:15.987081 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.987031 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:15.987081 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.987041 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:15.988561 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.988547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:15.988609 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.988572 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:15.989243 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.989227 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:15.989306 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.989261 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:15.989306 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:15.989277 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:16.026212 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.026191 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.028873 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.028852 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.049591 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.049565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.049677 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.049592 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.049677 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.049615 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc41cca5aaa38cfa8c691e2d5baf2778-config\") pod \"kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"dc41cca5aaa38cfa8c691e2d5baf2778\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.076949 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.076924 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 14:45:16.077030 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.076997 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bl97l" Oct 31 14:45:16.078080 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.078052 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 14:45:16.078080 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.078059 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 14:45:16.078080 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.078067 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-160.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 14:45:16.078080 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.078015 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-160.us-west-1.compute.internal.18e3a478e4d5108c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-160.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-160.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-160.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 14:45:15.765788812 +0000 UTC m=+0.405307038,LastTimestamp:2026-10-31 14:45:15.765788812 +0000 UTC m=+0.405307038,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-160.us-west-1.compute.internal,}" Oct 31 14:45:16.097983 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.097961 2578 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-160.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.150016 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.149991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.150111 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.150018 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.150111 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.150042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc41cca5aaa38cfa8c691e2d5baf2778-config\") pod \"kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"dc41cca5aaa38cfa8c691e2d5baf2778\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.150111 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.150095 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc41cca5aaa38cfa8c691e2d5baf2778-config\") pod \"kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"dc41cca5aaa38cfa8c691e2d5baf2778\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.150221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.150095 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.150221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.150118 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.189575 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.189557 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:16.242258 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.242169 2578 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-160.us-west-1.compute.internal\" not found" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.298954 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.298906 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:16.299939 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.299922 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:16.300034 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.299953 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:16.300034 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.299963 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:16.300034 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.299997 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.329351 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.329317 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.332450 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.332423 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.357957 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.357927 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:16.357957 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.357958 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-160.us-west-1.compute.internal\": node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:16.482075 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.482042 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:16.582241 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.582153 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:16.673136 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.673105 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 14:45:16.673666 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.673249 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 14:45:16.673666 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.673271 2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-160.us-west-1.compute.internal.18e3a478e4d573b5\": read tcp 10.0.137.160:38904->3.147.1.255:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-160.us-west-1.compute.internal.18e3a478e4d573b5 default 5051 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-160.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-160.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-160.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 14:45:15 +0000 UTC,LastTimestamp:2026-10-31 14:45:15.94061799 +0000 UTC m=+0.580136212,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-160.us-west-1.compute.internal,}" Oct 31 14:45:16.682404 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.682380 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:16.783205 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.783179 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:16.882597 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.882538 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:16.883255 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.883236 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:16.883467 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.883451 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:16.883528 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.883478 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:16.883528 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:16.883488 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:16.983341 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:16.983315 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:17.082482 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.082443 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 14:40:16 +0000 UTC" deadline="2028-08-08 02:48:30.31118299 +0000 UTC" Oct 31 14:45:17.082482 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.082477 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15516h3m13.228708908s" Oct 31 14:45:17.083775 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:17.083759 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:17.131079 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.131056 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:17.184429 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:17.184372 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:17.285162 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:17.285126 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:17.356317 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:17.356283 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc41cca5aaa38cfa8c691e2d5baf2778.slice/crio-a2dcbf4ff4819c9d38fda7b3267600e533cab5070b94bec430c7c78b813e7110 WatchSource:0}: Error finding container a2dcbf4ff4819c9d38fda7b3267600e533cab5070b94bec430c7c78b813e7110: Status 404 returned error can't find the container with id a2dcbf4ff4819c9d38fda7b3267600e533cab5070b94bec430c7c78b813e7110 Oct 31 14:45:17.365580 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.365565 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:45:17.374591 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:17.374571 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f83f99f29758da3777ab885397f404f.slice/crio-890268a700baeec6115805f613da701bbfed12978caee4e3e0c261f3ab393c86 WatchSource:0}: Error finding container 890268a700baeec6115805f613da701bbfed12978caee4e3e0c261f3ab393c86: Status 404 returned error can't find the container with id 890268a700baeec6115805f613da701bbfed12978caee4e3e0c261f3ab393c86 Oct 31 14:45:17.385749 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:17.385732 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:17.486424 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:17.486349 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:17.527392 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.527374 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:17.586632 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:17.586595 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:17.686752 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:17.686727 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:17.715886 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.715865 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:17.749049 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.749000 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 14:45:17.749049 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.749011 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:17.812786 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.812770 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:17.813617 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.813604 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:17.875007 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.874984 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:17.886824 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.886774 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"3f83f99f29758da3777ab885397f404f","Type":"ContainerStarted","Data":"890268a700baeec6115805f613da701bbfed12978caee4e3e0c261f3ab393c86"} Oct 31 14:45:17.887986 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.887965 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"dc41cca5aaa38cfa8c691e2d5baf2778","Type":"ContainerStarted","Data":"a2dcbf4ff4819c9d38fda7b3267600e533cab5070b94bec430c7c78b813e7110"} Oct 31 14:45:17.974193 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.974157 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 14:45:17.974340 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:17.974239 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xdf47" Oct 31 14:45:18.283161 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.283133 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:18.714611 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.714327 2578 apiserver.go:52] "Watching apiserver" Oct 31 14:45:18.879549 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.879514 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 14:45:18.879948 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.879919 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-t8nzm","kube-system/global-pull-secret-syncer-fjcpl","openshift-cluster-node-tuning-operator/tuned-wp5zq","openshift-dns/node-resolver-rqlqd","openshift-network-operator/iptables-alerter-gmn9f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-tl4sm","openshift-network-diagnostics/network-check-target-g9jmb","kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-nrbnb","openshift-ovn-kubernetes/ovnkube-node-7bxzn","kube-system/konnectivity-agent-tqg7k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp","openshift-image-registry/node-ca-r2dg7"] Oct 31 14:45:18.882706 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.882556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:18.882706 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:18.882645 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:18.884904 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.884876 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:18.887324 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.887300 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.890096 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.889756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:18.892586 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.892554 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:18.892691 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:18.892655 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:18.895225 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.895208 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.897971 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.897947 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:18.900575 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.900517 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:18.903130 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.903097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.905749 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.905681 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:18.908277 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.908254 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:18.911382 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.911363 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:18.911478 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:18.911428 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:18.949502 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.949484 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 14:45:18.965785 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.965686 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 14:45:18.965899 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.965854 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 14:45:18.965962 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.965927 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 14:45:18.966017 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.965996 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 14:45:18.966076 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966047 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 14:45:18.966126 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966094 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 14:45:18.966214 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-systemd\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.966271 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966229 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-tuned\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.966271 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966286 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966325 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbc7\" (UniqueName: \"kubernetes.io/projected/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-kube-api-access-mgbc7\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966349 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt6ss\" (UniqueName: \"kubernetes.io/projected/5fabf907-4df5-4334-86e9-369dd561a536-kube-api-access-rt6ss\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-kubelet\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966397 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-log-socket\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966442 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-env-overrides\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:18.966476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966479 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjpl\" (UniqueName: \"kubernetes.io/projected/f73cb9cb-94d1-453f-a199-3d8bd48a2079-kube-api-access-jsjpl\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966518 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-daemon-config\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966535 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92v4c\" (UniqueName: \"kubernetes.io/projected/e7903225-51db-41c7-a143-ea5b3b6a9fbe-kube-api-access-92v4c\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966552 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwktb\" (UniqueName: \"kubernetes.io/projected/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-kube-api-access-fwktb\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf78675e-1580-42ee-a251-9f03b56179c7-iptables-alerter-script\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-modprobe-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0eeb3c8b-578d-4a72-af8f-9c8c53887516-konnectivity-ca\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966657 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-host\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966672 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-config\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966691 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-kubelet\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966706 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-var-lib-kubelet\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966740 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc2lq\" (UniqueName: \"kubernetes.io/projected/254c6437-651d-484e-bbcc-6d6e393b88f7-kube-api-access-mc2lq\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0eeb3c8b-578d-4a72-af8f-9c8c53887516-agent-certs\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-k8s-cni-cncf-io\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.967032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-system-cni-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966882 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-conf\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8glxg\" (UniqueName: \"kubernetes.io/projected/ba001443-fbac-4b4f-be07-179fcdefa740-kube-api-access-8glxg\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.966969 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-device-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967005 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-dbus\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967026 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-bin\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-netd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967074 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967098 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967121 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-sys\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967141 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-lib-modules\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-system-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967173 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-socket-dir-parent\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-netns\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.967565 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967205 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-bin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967225 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-serviceca\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967274 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-multus-certs\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967300 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysconfig\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967315 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-tmp\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-netns\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967348 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967365 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srfm\" (UniqueName: \"kubernetes.io/projected/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-kube-api-access-4srfm\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-hostroot\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967405 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgrfh\" (UniqueName: \"kubernetes.io/projected/cf78675e-1580-42ee-a251-9f03b56179c7-kube-api-access-cgrfh\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cnibin\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967456 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-kubernetes\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967472 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-systemd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-ovn\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-node-log\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-script-lib\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cnibin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967589 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-multus\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967614 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba001443-fbac-4b4f-be07-179fcdefa740-hosts-file\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-systemd-units\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967662 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-etc-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.967681 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovn-node-metrics-cert\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-run\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-host\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968282 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968308 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba001443-fbac-4b4f-be07-179fcdefa740-tmp-dir\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968335 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-slash\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-etc-kubernetes\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-kubelet-config\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-var-lib-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968444 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cni-binary-copy\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.968882 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968470 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-conf-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:18.969520 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968497 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf78675e-1580-42ee-a251-9f03b56179c7-host-slash\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:18.969520 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.968525 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:18.970017 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.969996 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 14:45:18.970137 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.970084 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\"" Oct 31 14:45:18.970248 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.970227 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\"" Oct 31 14:45:18.978232 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.978179 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 14:40:18 +0000 UTC" deadline="2028-07-27 03:13:25.25946918 +0000 UTC" Oct 31 14:45:18.978232 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.978233 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15228h28m6.281240787s" Oct 31 14:45:18.993254 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.993234 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 14:45:18.993415 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.993386 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 14:45:18.993476 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.993432 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\"" Oct 31 14:45:18.993646 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.993627 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:18.993752 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.993724 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 14:45:18.993900 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.993872 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\"" Oct 31 14:45:18.994062 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.994042 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 14:45:18.994168 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.994155 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\"" Oct 31 14:45:18.994400 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.994381 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\"" Oct 31 14:45:18.994512 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.994452 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 14:45:18.994512 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.994510 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 14:45:18.994641 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.994571 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 14:45:18.994696 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:18.994668 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 14:45:19.019439 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.019415 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 14:45:19.019563 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.019455 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\"" Oct 31 14:45:19.019563 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.019470 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 14:45:19.023630 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.023609 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\"" Oct 31 14:45:19.024043 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.024020 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 14:45:19.024127 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.024082 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\"" Oct 31 14:45:19.047064 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.047022 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:19.047064 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.047048 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 14:45:19.047273 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.047110 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:19.047273 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.047150 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 14:45:19.047273 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.047171 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 14:45:19.047273 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.047177 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:19.047273 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.047217 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 14:45:19.047273 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.047230 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 14:45:19.069041 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069002 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-netns\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069149 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069048 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069149 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srfm\" (UniqueName: \"kubernetes.io/projected/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-kube-api-access-4srfm\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:19.069149 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-hostroot\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.069149 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069132 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgrfh\" (UniqueName: \"kubernetes.io/projected/cf78675e-1580-42ee-a251-9f03b56179c7-kube-api-access-cgrfh\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:19.069149 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069132 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-netns\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069369 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cnibin\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.069369 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-kubernetes\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.069369 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069218 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:19.069369 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-systemd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069369 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069265 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-ovn\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069369 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069279 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-node-log\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069369 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069300 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-script-lib\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069369 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cnibin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.069369 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069346 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-multus\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069372 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba001443-fbac-4b4f-be07-179fcdefa740-hosts-file\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069397 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-systemd-units\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069422 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-etc-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069445 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovn-node-metrics-cert\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069461 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-run\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069477 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-host\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069519 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069532 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-host\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069552 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba001443-fbac-4b4f-be07-179fcdefa740-tmp-dir\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069578 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-slash\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069583 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-hostroot\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069604 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-etc-kubernetes\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-kubelet-config\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069659 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-var-lib-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cni-binary-copy\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069709 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-conf-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.069805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf78675e-1580-42ee-a251-9f03b56179c7-host-slash\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069886 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cnibin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069940 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-multus\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.069995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba001443-fbac-4b4f-be07-179fcdefa740-hosts-file\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-systemd-units\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070083 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-etc-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070398 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-script-lib\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070379 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.070445 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-systemd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070507 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-ovn\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.070509 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret podName:8c8ff594-7d79-4f56-9eb4-a7af14808c03 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:19.570490075 +0000 UTC m=+4.210008288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret") pod "global-pull-secret-syncer-fjcpl" (UID: "8c8ff594-7d79-4f56-9eb4-a7af14808c03") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070546 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cnibin\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070554 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-node-log\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.070629 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070605 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-kubernetes\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.071221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-run\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.071221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070660 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-etc-kubernetes\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.071221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.070993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba001443-fbac-4b4f-be07-179fcdefa740-tmp-dir\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:19.071221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071058 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-var-lib-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.071221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071107 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-kubelet-config\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:19.071221 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071181 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.071399 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071230 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-slash\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.071399 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071259 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-conf-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.071399 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071359 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf78675e-1580-42ee-a251-9f03b56179c7-host-slash\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:19.071805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071763 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cni-binary-copy\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.071920 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.071965 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-systemd\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.071965 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-tuned\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.072066 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.071984 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.072066 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072014 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.072066 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.072220 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072066 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbc7\" (UniqueName: \"kubernetes.io/projected/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-kube-api-access-mgbc7\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:19.072220 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072094 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:19.072220 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6ss\" (UniqueName: \"kubernetes.io/projected/5fabf907-4df5-4334-86e9-369dd561a536-kube-api-access-rt6ss\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.072220 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-kubelet\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.072220 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-log-socket\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.072461 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.072461 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072266 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-env-overrides\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.072461 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.072461 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072356 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjpl\" (UniqueName: \"kubernetes.io/projected/f73cb9cb-94d1-453f-a199-3d8bd48a2079-kube-api-access-jsjpl\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.072461 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:19.072461 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-daemon-config\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.072461 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072429 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92v4c\" (UniqueName: \"kubernetes.io/projected/e7903225-51db-41c7-a143-ea5b3b6a9fbe-kube-api-access-92v4c\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.072461 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072456 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwktb\" (UniqueName: \"kubernetes.io/projected/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-kube-api-access-fwktb\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072483 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf78675e-1580-42ee-a251-9f03b56179c7-iptables-alerter-script\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072508 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-modprobe-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0eeb3c8b-578d-4a72-af8f-9c8c53887516-konnectivity-ca\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-host\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072587 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-config\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-kubelet\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072642 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-var-lib-kubelet\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2lq\" (UniqueName: \"kubernetes.io/projected/254c6437-651d-484e-bbcc-6d6e393b88f7-kube-api-access-mc2lq\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0eeb3c8b-578d-4a72-af8f-9c8c53887516-agent-certs\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072745 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-k8s-cni-cncf-io\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-system-cni-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072817 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-conf\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072864 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8glxg\" (UniqueName: \"kubernetes.io/projected/ba001443-fbac-4b4f-be07-179fcdefa740-kube-api-access-8glxg\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072888 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.072930 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072911 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-device-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-dbus\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-bin\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-netd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.072997 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073059 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-sys\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073084 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-lib-modules\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073115 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-system-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-socket-dir-parent\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073169 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-netns\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073197 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-bin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073250 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-serviceca\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-multus-certs\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysconfig\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.073723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073336 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-tmp\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073810 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-var-lib-kubelet\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.073960 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.074020 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-systemd\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.074291 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-netd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.074553 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovn-node-metrics-cert\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.074647 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-socket-dir-parent\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.074756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.074874 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.074928 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-sys\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.075019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-lib-modules\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.075074 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-system-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.075539 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.075654 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.075727 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.075744 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.075789 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-k8s-cni-cncf-io\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.086756 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.075831 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.075843 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-system-cni-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.076357 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.076425 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.076472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-log-socket\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.076511 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-conf\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.076808 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.076863 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-env-overrides\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.076873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-device-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.076976 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-dbus\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.077027 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-bin\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.077066 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-kubelet\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.077111 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-host\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.077196 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-modprobe-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.077526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-kubelet\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.077612 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.077724 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.077738 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-netns\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.087375 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.077795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-bin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.077846 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:19.577826221 +0000 UTC m=+4.217344446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.077845 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-multus-certs\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.078017 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysconfig\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.078472 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf78675e-1580-42ee-a251-9f03b56179c7-iptables-alerter-script\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.078911 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0eeb3c8b-578d-4a72-af8f-9c8c53887516-agent-certs\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.079438 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-daemon-config\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.079657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-tuned\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.079939 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-tmp\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.079957 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-serviceca\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.079962 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0eeb3c8b-578d-4a72-af8f-9c8c53887516-konnectivity-ca\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:19.088097 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.079987 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-config\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.130660 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.130630 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgrfh\" (UniqueName: \"kubernetes.io/projected/cf78675e-1580-42ee-a251-9f03b56179c7-kube-api-access-cgrfh\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:19.130996 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.130972 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srfm\" (UniqueName: \"kubernetes.io/projected/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-kube-api-access-4srfm\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:19.134126 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.134094 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:19.134126 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.134121 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:19.134291 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.134135 2578 projected.go:196] Error preparing data for projected volume kube-api-access-txsxx for pod openshift-network-diagnostics/network-check-target-g9jmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:19.134291 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.134198 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx podName:da21ed35-d5b8-4ef5-a6db-69b066ce2f6c nodeName:}" failed. No retries permitted until 2026-10-31 14:45:19.63417961 +0000 UTC m=+4.273697836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-txsxx" (UniqueName: "kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx") pod "network-check-target-g9jmb" (UID: "da21ed35-d5b8-4ef5-a6db-69b066ce2f6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:19.135752 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.135703 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2lq\" (UniqueName: \"kubernetes.io/projected/254c6437-651d-484e-bbcc-6d6e393b88f7-kube-api-access-mc2lq\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.137469 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.137188 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbc7\" (UniqueName: \"kubernetes.io/projected/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-kube-api-access-mgbc7\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:19.137808 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.137756 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6ss\" (UniqueName: \"kubernetes.io/projected/5fabf907-4df5-4334-86e9-369dd561a536-kube-api-access-rt6ss\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.138993 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.138930 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92v4c\" (UniqueName: \"kubernetes.io/projected/e7903225-51db-41c7-a143-ea5b3b6a9fbe-kube-api-access-92v4c\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.139452 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.139427 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8glxg\" (UniqueName: \"kubernetes.io/projected/ba001443-fbac-4b4f-be07-179fcdefa740-kube-api-access-8glxg\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:19.140797 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.140773 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjpl\" (UniqueName: \"kubernetes.io/projected/f73cb9cb-94d1-453f-a199-3d8bd48a2079-kube-api-access-jsjpl\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.141258 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.141176 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwktb\" (UniqueName: \"kubernetes.io/projected/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-kube-api-access-fwktb\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.199926 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.199878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:19.213369 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:19.213324 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a602e7c_a12e_44d4_8dbe_a7a958e867b5.slice/crio-de76c03d0de7ecbbd90d0570269358d81fc9f7b3e609924b2517643beaa13e86 WatchSource:0}: Error finding container de76c03d0de7ecbbd90d0570269358d81fc9f7b3e609924b2517643beaa13e86: Status 404 returned error can't find the container with id de76c03d0de7ecbbd90d0570269358d81fc9f7b3e609924b2517643beaa13e86 Oct 31 14:45:19.222207 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.222184 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:19.226416 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.226392 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t8nzm" Oct 31 14:45:19.229247 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.229111 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:19.234543 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:19.234340 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba001443_fbac_4b4f_be07_179fcdefa740.slice/crio-18d33bcf35fdba83fdf3606b5c1b547b99f7b533c9a0782f36bbd42b2a73711f WatchSource:0}: Error finding container 18d33bcf35fdba83fdf3606b5c1b547b99f7b533c9a0782f36bbd42b2a73711f: Status 404 returned error can't find the container with id 18d33bcf35fdba83fdf3606b5c1b547b99f7b533c9a0782f36bbd42b2a73711f Oct 31 14:45:19.234543 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.234402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:19.239455 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:19.238989 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f49f18_fdb2_4d17_ae2e_95ec4ea53b14.slice/crio-8dd0f1b62270fd33b773b294481f95357d38f9d07a80c0157d3aaa6fb87bf512 WatchSource:0}: Error finding container 8dd0f1b62270fd33b773b294481f95357d38f9d07a80c0157d3aaa6fb87bf512: Status 404 returned error can't find the container with id 8dd0f1b62270fd33b773b294481f95357d38f9d07a80c0157d3aaa6fb87bf512 Oct 31 14:45:19.245806 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:19.245760 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73cb9cb_94d1_453f_a199_3d8bd48a2079.slice/crio-7e6096b961087455db523a013075d0205c051e01c9ca8d993bfa988000d8417d WatchSource:0}: Error finding container 7e6096b961087455db523a013075d0205c051e01c9ca8d993bfa988000d8417d: Status 404 returned error can't find the container with id 7e6096b961087455db523a013075d0205c051e01c9ca8d993bfa988000d8417d Oct 31 14:45:19.251462 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:19.251437 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf78675e_1580_42ee_a251_9f03b56179c7.slice/crio-91e1571d431617f062f18653cd76d5a136a70ba466ed9c65873905303002ff06 WatchSource:0}: Error finding container 91e1571d431617f062f18653cd76d5a136a70ba466ed9c65873905303002ff06: Status 404 returned error can't find the container with id 91e1571d431617f062f18653cd76d5a136a70ba466ed9c65873905303002ff06 Oct 31 14:45:19.260211 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.260187 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:19.268704 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.268682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:19.269484 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:19.269459 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254c6437_651d_484e_bbcc_6d6e393b88f7.slice/crio-e19832601d4f2c09e93564fdd89053bd20e180478f3dbec8555f988bab5e6b80 WatchSource:0}: Error finding container e19832601d4f2c09e93564fdd89053bd20e180478f3dbec8555f988bab5e6b80: Status 404 returned error can't find the container with id e19832601d4f2c09e93564fdd89053bd20e180478f3dbec8555f988bab5e6b80 Oct 31 14:45:19.270602 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.270584 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:19.279888 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.279863 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:19.283953 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:19.283921 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7903225_51db_41c7_a143_ea5b3b6a9fbe.slice/crio-88b513be41a5c06f48052c4f5e97df1c7276061b7c4421e91c827a60fbfa6bb6 WatchSource:0}: Error finding container 88b513be41a5c06f48052c4f5e97df1c7276061b7c4421e91c827a60fbfa6bb6: Status 404 returned error can't find the container with id 88b513be41a5c06f48052c4f5e97df1c7276061b7c4421e91c827a60fbfa6bb6 Oct 31 14:45:19.284229 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:19.284182 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eeb3c8b_578d_4a72_af8f_9c8c53887516.slice/crio-eb408cc802b20179cd8626d72b3cf38f69dc200174df8ffb547129e547fcf498 WatchSource:0}: Error finding container eb408cc802b20179cd8626d72b3cf38f69dc200174df8ffb547129e547fcf498: Status 404 returned error can't find the container with id eb408cc802b20179cd8626d72b3cf38f69dc200174df8ffb547129e547fcf498 Oct 31 14:45:19.320077 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:19.320055 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fabf907_4df5_4334_86e9_369dd561a536.slice/crio-0e82938e30bb9ff73af930caac8ca143bc69b037e0bebfd2a7c7c6e820168d4d WatchSource:0}: Error finding container 0e82938e30bb9ff73af930caac8ca143bc69b037e0bebfd2a7c7c6e820168d4d: Status 404 returned error can't find the container with id 0e82938e30bb9ff73af930caac8ca143bc69b037e0bebfd2a7c7c6e820168d4d Oct 31 14:45:19.577075 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.576930 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:19.577169 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.577087 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:19.577169 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.577149 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret podName:8c8ff594-7d79-4f56-9eb4-a7af14808c03 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:20.577135264 +0000 UTC m=+5.216653491 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret") pod "global-pull-secret-syncer-fjcpl" (UID: "8c8ff594-7d79-4f56-9eb4-a7af14808c03") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:19.678412 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.678387 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:19.678498 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.678424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:19.678550 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.678521 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:19.678587 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.678567 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:20.678554151 +0000 UTC m=+5.318072362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:19.678587 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.678574 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:19.678675 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.678603 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:19.678675 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.678618 2578 projected.go:196] Error preparing data for projected volume kube-api-access-txsxx for pod openshift-network-diagnostics/network-check-target-g9jmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:19.678749 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:19.678681 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx podName:da21ed35-d5b8-4ef5-a6db-69b066ce2f6c nodeName:}" failed. No retries permitted until 2026-10-31 14:45:20.678662111 +0000 UTC m=+5.318180336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-txsxx" (UniqueName: "kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx") pod "network-check-target-g9jmb" (UID: "da21ed35-d5b8-4ef5-a6db-69b066ce2f6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:19.892283 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.892173 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"dc41cca5aaa38cfa8c691e2d5baf2778","Type":"ContainerStarted","Data":"5d8421d1c7db655639c0d72a3d183f03803926060f363bf56b53d22c03eb0507"} Oct 31 14:45:19.893529 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.893504 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tqg7k" event={"ID":"0eeb3c8b-578d-4a72-af8f-9c8c53887516","Type":"ContainerStarted","Data":"eb408cc802b20179cd8626d72b3cf38f69dc200174df8ffb547129e547fcf498"} Oct 31 14:45:19.894508 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.894483 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" event={"ID":"254c6437-651d-484e-bbcc-6d6e393b88f7","Type":"ContainerStarted","Data":"e19832601d4f2c09e93564fdd89053bd20e180478f3dbec8555f988bab5e6b80"} Oct 31 14:45:19.895516 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.895498 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmn9f" event={"ID":"cf78675e-1580-42ee-a251-9f03b56179c7","Type":"ContainerStarted","Data":"91e1571d431617f062f18653cd76d5a136a70ba466ed9c65873905303002ff06"} Oct 31 14:45:19.896549 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.896530 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" event={"ID":"f73cb9cb-94d1-453f-a199-3d8bd48a2079","Type":"ContainerStarted","Data":"7e6096b961087455db523a013075d0205c051e01c9ca8d993bfa988000d8417d"} Oct 31 14:45:19.897605 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.897586 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rqlqd" event={"ID":"ba001443-fbac-4b4f-be07-179fcdefa740","Type":"ContainerStarted","Data":"18d33bcf35fdba83fdf3606b5c1b547b99f7b533c9a0782f36bbd42b2a73711f"} Oct 31 14:45:19.898599 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.898581 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" event={"ID":"5fabf907-4df5-4334-86e9-369dd561a536","Type":"ContainerStarted","Data":"0e82938e30bb9ff73af930caac8ca143bc69b037e0bebfd2a7c7c6e820168d4d"} Oct 31 14:45:19.899738 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.899699 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" event={"ID":"e7903225-51db-41c7-a143-ea5b3b6a9fbe","Type":"ContainerStarted","Data":"88b513be41a5c06f48052c4f5e97df1c7276061b7c4421e91c827a60fbfa6bb6"} Oct 31 14:45:19.900841 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.900823 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t8nzm" event={"ID":"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14","Type":"ContainerStarted","Data":"8dd0f1b62270fd33b773b294481f95357d38f9d07a80c0157d3aaa6fb87bf512"} Oct 31 14:45:19.901924 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.901902 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r2dg7" event={"ID":"5a602e7c-a12e-44d4-8dbe-a7a958e867b5","Type":"ContainerStarted","Data":"de76c03d0de7ecbbd90d0570269358d81fc9f7b3e609924b2517643beaa13e86"} Oct 31 14:45:19.903250 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.903230 2578 generic.go:356] "Generic (PLEG): container finished" podID="3f83f99f29758da3777ab885397f404f" containerID="4c8ddabcd862341a9cf446f6e61062a43a3b7a27a146698c3f5f2a008046f96a" exitCode=0 Oct 31 14:45:19.903326 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.903256 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"3f83f99f29758da3777ab885397f404f","Type":"ContainerDied","Data":"4c8ddabcd862341a9cf446f6e61062a43a3b7a27a146698c3f5f2a008046f96a"} Oct 31 14:45:19.978410 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.978366 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 14:40:18 +0000 UTC" deadline="2028-04-26 22:49:01.32448124 +0000 UTC" Oct 31 14:45:19.978410 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:19.978407 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13040h3m41.346077089s" Oct 31 14:45:20.585673 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:20.585636 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:20.585911 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.585883 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:20.585979 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.585951 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret podName:8c8ff594-7d79-4f56-9eb4-a7af14808c03 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:22.585931599 +0000 UTC m=+7.225449825 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret") pod "global-pull-secret-syncer-fjcpl" (UID: "8c8ff594-7d79-4f56-9eb4-a7af14808c03") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:20.687391 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:20.686892 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:20.687391 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:20.686945 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:20.687391 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.687144 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:20.687391 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.687207 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:22.687187174 +0000 UTC m=+7.326705405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:20.687391 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.687302 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:20.687391 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.687315 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:20.687391 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.687329 2578 projected.go:196] Error preparing data for projected volume kube-api-access-txsxx for pod openshift-network-diagnostics/network-check-target-g9jmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:20.687391 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.687366 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx podName:da21ed35-d5b8-4ef5-a6db-69b066ce2f6c nodeName:}" failed. No retries permitted until 2026-10-31 14:45:22.687353893 +0000 UTC m=+7.326872109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-txsxx" (UniqueName: "kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx") pod "network-check-target-g9jmb" (UID: "da21ed35-d5b8-4ef5-a6db-69b066ce2f6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:20.883039 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:20.882969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:20.883214 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.883089 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:20.883214 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:20.883206 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:20.883324 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.883300 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:20.883400 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:20.883385 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:20.883513 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:20.883494 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:20.937644 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:20.937606 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"3f83f99f29758da3777ab885397f404f","Type":"ContainerStarted","Data":"f198c843599aa8213bc48ba07657e9dd244053fd10032c36643b966f9ae2bda6"} Oct 31 14:45:21.058501 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:21.058435 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" podStartSLOduration=3.058416153 podStartE2EDuration="3.058416153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:20.897659734 +0000 UTC m=+5.537177963" watchObservedRunningTime="2026-10-31 14:45:21.058416153 +0000 UTC m=+5.697934388" Oct 31 14:45:22.605554 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:22.605278 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:22.605554 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.605438 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:22.605554 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.605495 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret podName:8c8ff594-7d79-4f56-9eb4-a7af14808c03 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:26.605477725 +0000 UTC m=+11.244995942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret") pod "global-pull-secret-syncer-fjcpl" (UID: "8c8ff594-7d79-4f56-9eb4-a7af14808c03") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:22.707977 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:22.707404 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:22.707977 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:22.707452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:22.707977 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.707573 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:22.707977 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.707632 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:26.707612909 +0000 UTC m=+11.347131121 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:22.707977 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.707717 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:22.707977 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.707730 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:22.707977 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.707744 2578 projected.go:196] Error preparing data for projected volume kube-api-access-txsxx for pod openshift-network-diagnostics/network-check-target-g9jmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:22.707977 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.707799 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx podName:da21ed35-d5b8-4ef5-a6db-69b066ce2f6c nodeName:}" failed. No retries permitted until 2026-10-31 14:45:26.707767401 +0000 UTC m=+11.347285628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-txsxx" (UniqueName: "kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx") pod "network-check-target-g9jmb" (UID: "da21ed35-d5b8-4ef5-a6db-69b066ce2f6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:22.883409 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:22.883330 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:22.883577 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.883471 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:22.883577 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:22.883552 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:22.883692 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.883644 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:22.883765 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:22.883706 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:22.883860 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:22.883807 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:22.955818 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:22.955759 2578 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="ca0aae7cecffe1f77ad9e3e6d9053f589c8d6e644adc8e5eb734541f1ce6ea63" exitCode=0 Oct 31 14:45:22.955991 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:22.955836 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" event={"ID":"e7903225-51db-41c7-a143-ea5b3b6a9fbe","Type":"ContainerDied","Data":"ca0aae7cecffe1f77ad9e3e6d9053f589c8d6e644adc8e5eb734541f1ce6ea63"} Oct 31 14:45:23.080887 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:23.080798 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" podStartSLOduration=5.080779595 podStartE2EDuration="5.080779595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:21.059651521 +0000 UTC m=+5.699169752" watchObservedRunningTime="2026-10-31 14:45:23.080779595 +0000 UTC m=+7.720297827" Oct 31 14:45:23.961844 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:23.961554 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tqg7k" event={"ID":"0eeb3c8b-578d-4a72-af8f-9c8c53887516","Type":"ContainerStarted","Data":"f72c8c39e83886133a1b30c96bca93bf7c2b28ba24e3719ada2b41d00c0bdae8"} Oct 31 14:45:24.884159 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:24.883299 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:24.884159 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:24.883451 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:24.884159 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:24.883878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:24.884159 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:24.883975 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:24.884159 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:24.884041 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:24.884159 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:24.884113 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:24.969822 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:24.969667 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-r2dg7" event={"ID":"5a602e7c-a12e-44d4-8dbe-a7a958e867b5","Type":"ContainerStarted","Data":"93e2e268c8eef2688bf25aaa4c54f524515e96551b6cce850d0af0b56a056d38"} Oct 31 14:45:25.090880 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:25.090032 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tqg7k" podStartSLOduration=3.6804379430000003 podStartE2EDuration="8.090012321s" totalImagesPullingTime="4.409574378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:24.097200691 +0000 UTC m=+8.736718925" watchObservedRunningTime="2026-10-31 14:45:25.090012321 +0000 UTC m=+9.729530556" Oct 31 14:45:26.642540 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:26.642503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:26.643021 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.642723 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:26.643021 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.642795 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret podName:8c8ff594-7d79-4f56-9eb4-a7af14808c03 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:34.642777803 +0000 UTC m=+19.282296028 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret") pod "global-pull-secret-syncer-fjcpl" (UID: "8c8ff594-7d79-4f56-9eb4-a7af14808c03") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:26.743796 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:26.743497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:26.743796 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:26.743548 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:26.743796 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.743668 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:26.743796 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.743675 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:26.743796 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.743720 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:26.743796 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.743734 2578 projected.go:196] Error preparing data for projected volume kube-api-access-txsxx for pod openshift-network-diagnostics/network-check-target-g9jmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:26.743796 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.743752 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:34.743732545 +0000 UTC m=+19.383250756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:26.743796 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.743782 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx podName:da21ed35-d5b8-4ef5-a6db-69b066ce2f6c nodeName:}" failed. No retries permitted until 2026-10-31 14:45:34.743766736 +0000 UTC m=+19.383284944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-txsxx" (UniqueName: "kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx") pod "network-check-target-g9jmb" (UID: "da21ed35-d5b8-4ef5-a6db-69b066ce2f6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:26.882992 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:26.882957 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:26.883159 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.883094 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:26.883523 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:26.883499 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:26.883641 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.883618 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:26.883723 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:26.883679 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:26.883790 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:26.883774 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:26.980253 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:26.980083 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" event={"ID":"5fabf907-4df5-4334-86e9-369dd561a536","Type":"ContainerStarted","Data":"2ab13ea6368327998acfd948e27b8f907dde9cf56de3def47a199058389deb66"} Oct 31 14:45:27.986907 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:27.986873 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t8nzm" event={"ID":"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14","Type":"ContainerStarted","Data":"2b33c9d3a027504e2527bb1cfd800dfa91e7b4be742087ce8e10ec503e44db73"} Oct 31 14:45:28.136057 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.135805 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-r2dg7" podStartSLOduration=6.906465976 podStartE2EDuration="12.135785305s" totalImagesPullingTime="5.229319329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:25.090846234 +0000 UTC m=+9.730364469" watchObservedRunningTime="2026-10-31 14:45:28.135785305 +0000 UTC m=+12.775303552" Oct 31 14:45:28.136174 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.136140 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-t8nzm" podStartSLOduration=2.916621359 podStartE2EDuration="11.13612984s" totalImagesPullingTime="8.219508481s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:28.10223287 +0000 UTC m=+12.741751101" watchObservedRunningTime="2026-10-31 14:45:28.13612984 +0000 UTC m=+12.775648071" Oct 31 14:45:28.164008 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.163983 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 14:45:28.174711 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.174670 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:28.175438 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.175417 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:28.838887 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.838731 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T14:45:28.164003714Z","UUID":"310c7352-0aac-4067-8079-6738e878d77c","Handler":null,"Name":"","Endpoint":""} Oct 31 14:45:28.840333 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.840302 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 14:45:28.840333 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.840336 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 14:45:28.883080 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.883051 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:28.883256 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.883092 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:28.883256 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.883061 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:28.883256 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:28.883191 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:28.883449 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:28.883317 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:28.883449 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:28.883388 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:28.990691 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.990658 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rqlqd" event={"ID":"ba001443-fbac-4b4f-be07-179fcdefa740","Type":"ContainerStarted","Data":"2e5191e8da24293fdc2809d6dbf231d92140dd26d4ba6371665ecd983b5574a1"} Oct 31 14:45:28.992902 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.992849 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" event={"ID":"5fabf907-4df5-4334-86e9-369dd561a536","Type":"ContainerStarted","Data":"87d2714c2a767d56825931946ae4962ce32a51326c94876038530c406c4972c5"} Oct 31 14:45:28.996831 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.996805 2578 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="eaaecf8c2d27a32f0bd14d61466bc3505f554ac11242f7add7d40ff555ad55f6" exitCode=0 Oct 31 14:45:28.996905 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.996868 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" event={"ID":"e7903225-51db-41c7-a143-ea5b3b6a9fbe","Type":"ContainerDied","Data":"eaaecf8c2d27a32f0bd14d61466bc3505f554ac11242f7add7d40ff555ad55f6"} Oct 31 14:45:28.997552 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.997536 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:28.998191 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:28.998145 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:29.224185 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:29.224113 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-rqlqd" podStartSLOduration=4.452171815 podStartE2EDuration="13.224093265s" totalImagesPullingTime="8.77192145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:29.102665241 +0000 UTC m=+13.742183470" watchObservedRunningTime="2026-10-31 14:45:29.224093265 +0000 UTC m=+13.863611496" Oct 31 14:45:30.000391 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:30.000335 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" event={"ID":"254c6437-651d-484e-bbcc-6d6e393b88f7","Type":"ContainerStarted","Data":"91d946f05d18d236b97149511fa21f71e6a4f123054d9045dfd2f38fd7d04d2f"} Oct 31 14:45:30.002011 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:30.001975 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gmn9f" event={"ID":"cf78675e-1580-42ee-a251-9f03b56179c7","Type":"ContainerStarted","Data":"5b3b4ae80262568a7f54f55b3c2f7b663cfa2d927c983e105ead65900244bb43"} Oct 31 14:45:30.004359 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:30.004337 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" event={"ID":"5fabf907-4df5-4334-86e9-369dd561a536","Type":"ContainerStarted","Data":"b01c12ef62f664796f0229fd0e4e41565598d9932205c29bfcf981d3a70d4025"} Oct 31 14:45:30.113865 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:30.113813 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" podStartSLOduration=2.606367397 podStartE2EDuration="13.113795194s" totalImagesPullingTime="10.507427797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:30.113037515 +0000 UTC m=+14.752555758" watchObservedRunningTime="2026-10-31 14:45:30.113795194 +0000 UTC m=+14.753313427" Oct 31 14:45:30.230106 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:30.230054 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" podStartSLOduration=3.323321351 podStartE2EDuration="13.230038224s" totalImagesPullingTime="9.906716873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:30.229559298 +0000 UTC m=+14.869077528" watchObservedRunningTime="2026-10-31 14:45:30.230038224 +0000 UTC m=+14.869556451" Oct 31 14:45:30.341036 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:30.340982 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gmn9f" podStartSLOduration=4.721829834 podStartE2EDuration="13.340967273s" totalImagesPullingTime="8.619137439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:30.340305756 +0000 UTC m=+14.979823988" watchObservedRunningTime="2026-10-31 14:45:30.340967273 +0000 UTC m=+14.980485503" Oct 31 14:45:30.883248 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:30.883216 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:30.883400 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:30.883280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:30.883400 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:30.883386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:30.883524 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:30.883399 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:30.883524 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:30.883509 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:30.883637 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:30.883617 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:31.008355 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:31.008317 2578 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="186d3f3ace1ca444ff3eb348ef2e0d6d27f4a009fa4bee2e628819943ed3ae5e" exitCode=0 Oct 31 14:45:31.008903 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:31.008421 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" event={"ID":"e7903225-51db-41c7-a143-ea5b3b6a9fbe","Type":"ContainerDied","Data":"186d3f3ace1ca444ff3eb348ef2e0d6d27f4a009fa4bee2e628819943ed3ae5e"} Oct 31 14:45:32.883467 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:32.883258 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:32.884049 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:32.883302 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:32.884049 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:32.883572 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:32.884049 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:32.883637 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:32.884049 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:32.883335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:32.884049 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:32.883743 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:33.016178 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:33.016138 2578 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="71b9dd5c5daa65949e4e600d869e0357082bf9751bac65ca74780db23f51bb25" exitCode=0 Oct 31 14:45:33.016347 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:33.016206 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" event={"ID":"e7903225-51db-41c7-a143-ea5b3b6a9fbe","Type":"ContainerDied","Data":"71b9dd5c5daa65949e4e600d869e0357082bf9751bac65ca74780db23f51bb25"} Oct 31 14:45:34.705030 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:34.704850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:34.705347 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.704982 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:34.705347 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.705121 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret podName:8c8ff594-7d79-4f56-9eb4-a7af14808c03 nodeName:}" failed. No retries permitted until 2026-10-31 14:45:50.705103369 +0000 UTC m=+35.344621578 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret") pod "global-pull-secret-syncer-fjcpl" (UID: "8c8ff594-7d79-4f56-9eb4-a7af14808c03") : object "kube-system"/"original-pull-secret" not registered Oct 31 14:45:34.805869 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:34.805789 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:34.805869 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:34.805842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:34.806050 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.805951 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:34.806050 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.805979 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 14:45:34.806050 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.806000 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:50.805983201 +0000 UTC m=+35.445501408 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 14:45:34.806050 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.806006 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 14:45:34.806050 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.806020 2578 projected.go:196] Error preparing data for projected volume kube-api-access-txsxx for pod openshift-network-diagnostics/network-check-target-g9jmb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:34.806249 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.806068 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx podName:da21ed35-d5b8-4ef5-a6db-69b066ce2f6c nodeName:}" failed. No retries permitted until 2026-10-31 14:45:50.806052439 +0000 UTC m=+35.445570652 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-txsxx" (UniqueName: "kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx") pod "network-check-target-g9jmb" (UID: "da21ed35-d5b8-4ef5-a6db-69b066ce2f6c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 14:45:34.882398 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:34.882370 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:34.882398 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:34.882398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:34.882588 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.882484 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:34.882588 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:34.882512 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:34.882588 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.882575 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:34.882674 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:34.882618 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:35.024594 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:35.024563 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" event={"ID":"f73cb9cb-94d1-453f-a199-3d8bd48a2079","Type":"ContainerStarted","Data":"6476c97dbe1d975ffe8066961d0f38f8df9240f994c122605261acde8411bfbf"} Oct 31 14:45:35.024805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:35.024643 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" event={"ID":"f73cb9cb-94d1-453f-a199-3d8bd48a2079","Type":"ContainerStarted","Data":"63943b1d39141db8a362e2e0a5204202ba216bd2dfaca4e6e2cd59f39a081af0"} Oct 31 14:45:35.024805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:35.024678 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" event={"ID":"f73cb9cb-94d1-453f-a199-3d8bd48a2079","Type":"ContainerStarted","Data":"b3784f8f9a3d997b2d8c414fc4e1a05371230f4ae8016dc77c50d4a628142e6a"} Oct 31 14:45:35.024805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:35.024736 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" event={"ID":"f73cb9cb-94d1-453f-a199-3d8bd48a2079","Type":"ContainerStarted","Data":"892be85cf2a20ff41fbda219e299a9b0a9ae4dc7bf20819bfa455228084b6178"} Oct 31 14:45:35.024805 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:35.024768 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" event={"ID":"f73cb9cb-94d1-453f-a199-3d8bd48a2079","Type":"ContainerStarted","Data":"9773fffc2d07c60d264c5c6a0a3ed1edc83ace1764610efbd80b504e2fef9b1c"} Oct 31 14:45:35.024947 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:35.024811 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" event={"ID":"f73cb9cb-94d1-453f-a199-3d8bd48a2079","Type":"ContainerStarted","Data":"328f3b323f40fa4c186238c711bae78b0adc6dc0650d6016fe7f1ac994f9ba04"} Oct 31 14:45:35.027317 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:35.027291 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" event={"ID":"e7903225-51db-41c7-a143-ea5b3b6a9fbe","Type":"ContainerStarted","Data":"ac273263d9c1230c7f516193f8784248c26ace657dd1dc5f64d92a5e8c283708"} Oct 31 14:45:36.031539 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:36.031511 2578 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="ac273263d9c1230c7f516193f8784248c26ace657dd1dc5f64d92a5e8c283708" exitCode=0 Oct 31 14:45:36.031982 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:36.031547 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" event={"ID":"e7903225-51db-41c7-a143-ea5b3b6a9fbe","Type":"ContainerDied","Data":"ac273263d9c1230c7f516193f8784248c26ace657dd1dc5f64d92a5e8c283708"} Oct 31 14:45:36.883084 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:36.883056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:36.883215 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:36.883056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:36.883215 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:36.883166 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:36.883280 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:36.883063 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:36.883307 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:36.883227 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:36.883338 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:36.883301 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:37.036573 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:37.036497 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" event={"ID":"f73cb9cb-94d1-453f-a199-3d8bd48a2079","Type":"ContainerStarted","Data":"efa4edc2ffbad64df26d3b7af146fba291bce2e8e90525012571bc325c60f2d0"} Oct 31 14:45:37.039457 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:37.039436 2578 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="4804601d61918dacda801090bb8483ed4ce3e248926fc721bfa8a1b2eb2b2e32" exitCode=0 Oct 31 14:45:37.039579 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:37.039464 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" event={"ID":"e7903225-51db-41c7-a143-ea5b3b6a9fbe","Type":"ContainerDied","Data":"4804601d61918dacda801090bb8483ed4ce3e248926fc721bfa8a1b2eb2b2e32"} Oct 31 14:45:38.044456 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:38.044423 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" event={"ID":"e7903225-51db-41c7-a143-ea5b3b6a9fbe","Type":"ContainerStarted","Data":"335e5a43413ecb2f0a55fbb17f21511db41ecd737e9bdc6e3afb4ae97d2eafcb"} Oct 31 14:45:38.261770 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:38.261724 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-nrbnb" podStartSLOduration=9.336742187 podStartE2EDuration="21.26168864s" totalImagesPullingTime="11.924946453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:17 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:38.211605662 +0000 UTC m=+22.851123914" watchObservedRunningTime="2026-10-31 14:45:38.26168864 +0000 UTC m=+22.901206871" Oct 31 14:45:38.882493 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:38.882458 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:38.882493 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:38.882491 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:38.882725 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:38.882505 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:38.882725 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:38.882587 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:38.882725 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:38.882716 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:38.882855 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:38.882830 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:39.050290 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:39.050003 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" event={"ID":"f73cb9cb-94d1-453f-a199-3d8bd48a2079","Type":"ContainerStarted","Data":"436ca921ca5c04a7d84a373f6818cd2de0a124f06319d0b7f96c432818c2ae06"} Oct 31 14:45:39.317434 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:39.317322 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" podStartSLOduration=8.289589462 podStartE2EDuration="23.317310103s" totalImagesPullingTime="15.027720641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:39.267677416 +0000 UTC m=+23.907195646" watchObservedRunningTime="2026-10-31 14:45:39.317310103 +0000 UTC m=+23.956828345" Oct 31 14:45:40.053643 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:40.053613 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:40.054238 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:40.054227 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:40.054410 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:40.054400 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:40.085154 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:40.085018 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:40.086624 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:40.086595 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:40.882478 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:40.882445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:40.882655 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:40.882445 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:40.882655 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:40.882569 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:40.882655 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:40.882579 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:40.882846 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:40.882667 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:40.882846 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:40.882757 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:41.616949 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:41.616904 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fjcpl"] Oct 31 14:45:41.617423 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:41.617031 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:41.617423 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:41.617125 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:41.617762 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:41.617729 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tl4sm"] Oct 31 14:45:41.617903 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:41.617833 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:41.617967 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:41.617943 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:41.618411 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:41.618346 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g9jmb"] Oct 31 14:45:41.618569 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:41.618424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:41.618569 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:41.618495 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:42.882710 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:42.882671 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:42.883073 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:42.882823 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:43.882529 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:43.882489 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:43.882722 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:43.882650 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:43.882722 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:43.882715 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:43.883150 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:43.882818 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:44.883032 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:44.882951 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:44.883382 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:44.883059 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:45.883006 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:45.882969 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:45.883170 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:45.883141 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tl4sm" podUID="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" Oct 31 14:45:45.883554 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:45.883188 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:45.883554 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:45.883307 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g9jmb" podUID="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" Oct 31 14:45:46.882986 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:46.882960 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:46.883128 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:46.883052 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fjcpl" podUID="8c8ff594-7d79-4f56-9eb4-a7af14808c03" Oct 31 14:45:47.695079 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.694887 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeReady" Oct 31 14:45:47.695444 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.695160 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 14:45:47.847251 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.847219 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nxdlw"] Oct 31 14:45:47.865986 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.865963 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wvbrg"] Oct 31 14:45:47.866371 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.866132 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:47.884111 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.884089 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wvbrg"] Oct 31 14:45:47.884111 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.884100 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:47.884311 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.884119 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nxdlw"] Oct 31 14:45:47.884311 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.884176 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:47.884428 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.884401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:47.918597 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.918564 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 14:45:47.918597 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.918564 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 14:45:47.918808 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.918714 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\"" Oct 31 14:45:47.936373 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.936342 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 14:45:47.936469 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.936339 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\"" Oct 31 14:45:47.957103 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.957065 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 14:45:47.957307 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.957287 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 14:45:47.957390 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.957308 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 14:45:47.957390 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.957332 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 14:45:47.957390 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.957359 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\"" Oct 31 14:45:47.957390 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.957372 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 14:45:47.957512 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:47.957404 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\"" Oct 31 14:45:48.004654 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.004616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-config-volume\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.004654 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.004650 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.004919 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.004680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-tmp-dir\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.004919 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.004777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab9f675-4ee1-4226-a69d-2e362ccb753e-cert\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:48.004919 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.004808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqs5s\" (UniqueName: \"kubernetes.io/projected/9ab9f675-4ee1-4226-a69d-2e362ccb753e-kube-api-access-mqs5s\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:48.004919 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.004863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdc9\" (UniqueName: \"kubernetes.io/projected/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-kube-api-access-5mdc9\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.105390 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.105350 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdc9\" (UniqueName: \"kubernetes.io/projected/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-kube-api-access-5mdc9\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.105390 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.105399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-config-volume\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.105650 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.105415 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.105650 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.105438 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-tmp-dir\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.105650 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.105455 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab9f675-4ee1-4226-a69d-2e362ccb753e-cert\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:48.105650 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.105481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqs5s\" (UniqueName: \"kubernetes.io/projected/9ab9f675-4ee1-4226-a69d-2e362ccb753e-kube-api-access-mqs5s\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:48.105650 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:48.105541 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:48.105650 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:48.105611 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls podName:6a4f5abc-05a0-4bef-ac48-b926c6b54a6a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:48.605591473 +0000 UTC m=+33.245109696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls") pod "dns-default-nxdlw" (UID: "6a4f5abc-05a0-4bef-ac48-b926c6b54a6a") : secret "dns-default-metrics-tls" not found Oct 31 14:45:48.105949 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.105827 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-tmp-dir\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.106148 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.106129 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-config-volume\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.109675 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.109656 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab9f675-4ee1-4226-a69d-2e362ccb753e-cert\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:48.162822 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.162793 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdc9\" (UniqueName: \"kubernetes.io/projected/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-kube-api-access-5mdc9\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.163046 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.163023 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqs5s\" (UniqueName: \"kubernetes.io/projected/9ab9f675-4ee1-4226-a69d-2e362ccb753e-kube-api-access-mqs5s\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:48.221325 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.221239 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:48.609202 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.609113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:48.609351 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:48.609229 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:48.609351 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:48.609283 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls podName:6a4f5abc-05a0-4bef-ac48-b926c6b54a6a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:49.60927047 +0000 UTC m=+34.248788691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls") pod "dns-default-nxdlw" (UID: "6a4f5abc-05a0-4bef-ac48-b926c6b54a6a") : secret "dns-default-metrics-tls" not found Oct 31 14:45:48.618809 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.618776 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wvbrg"] Oct 31 14:45:48.622869 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:48.622845 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ab9f675_4ee1_4226_a69d_2e362ccb753e.slice/crio-874bf9688925c0c30d84e9717df36a5bb5a8d10fd9606c3afc8fd778b670cb5a WatchSource:0}: Error finding container 874bf9688925c0c30d84e9717df36a5bb5a8d10fd9606c3afc8fd778b670cb5a: Status 404 returned error can't find the container with id 874bf9688925c0c30d84e9717df36a5bb5a8d10fd9606c3afc8fd778b670cb5a Oct 31 14:45:48.882787 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.882717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:48.935186 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.935159 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 14:45:48.935306 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:48.935259 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\"" Oct 31 14:45:49.071116 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:49.071088 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wvbrg" event={"ID":"9ab9f675-4ee1-4226-a69d-2e362ccb753e","Type":"ContainerStarted","Data":"874bf9688925c0c30d84e9717df36a5bb5a8d10fd9606c3afc8fd778b670cb5a"} Oct 31 14:45:49.615754 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:49.615717 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:49.616019 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:49.615853 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:49.616019 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:49.615933 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls podName:6a4f5abc-05a0-4bef-ac48-b926c6b54a6a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.615914274 +0000 UTC m=+36.255432507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls") pod "dns-default-nxdlw" (UID: "6a4f5abc-05a0-4bef-ac48-b926c6b54a6a") : secret "dns-default-metrics-tls" not found Oct 31 14:45:50.160910 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.160878 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zlmzj"] Oct 31 14:45:50.178463 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.178435 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.221389 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.221363 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zlmzj"] Oct 31 14:45:50.270248 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.270220 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 14:45:50.270383 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.270262 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\"" Oct 31 14:45:50.270383 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.270320 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 14:45:50.270383 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.270349 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 14:45:50.270494 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.270428 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 14:45:50.320940 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.320903 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-data-volume\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.321107 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.320961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8tm\" (UniqueName: \"kubernetes.io/projected/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-api-access-gb8tm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.321160 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.321097 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.321160 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.321134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-crio-socket\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.321241 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.321214 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.422236 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.422148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.422236 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.422187 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-crio-socket\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.422236 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.422219 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.422498 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.422295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-data-volume\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.422498 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.422335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8tm\" (UniqueName: \"kubernetes.io/projected/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-api-access-gb8tm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.422498 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.422365 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:50.422498 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.422438 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-crio-socket\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.422498 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.422450 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:50.922428638 +0000 UTC m=+35.561946863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : configmap "kube-rbac-proxy" not found Oct 31 14:45:50.422498 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.422361 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:50.422814 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.422534 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:50.922521317 +0000 UTC m=+35.562039529 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:50.422814 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.422652 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-data-volume\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.480178 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.480123 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8tm\" (UniqueName: \"kubernetes.io/projected/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-api-access-gb8tm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.725428 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.725391 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:50.727952 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.727923 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:50.826210 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.826162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:50.826379 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.826217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:50.826379 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.826319 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:45:50.826379 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.826374 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:46:22.826361426 +0000 UTC m=+67.465879634 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : secret "metrics-daemon-secret" not found Oct 31 14:45:50.829102 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.829079 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:50.927006 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.926977 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.927151 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.927080 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:50.927151 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.927105 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:50.927267 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.927173 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.927158388 +0000 UTC m=+36.566676610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : configmap "kube-rbac-proxy" not found Oct 31 14:45:50.927267 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.927195 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:50.927267 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:50.927248 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:51.927232508 +0000 UTC m=+36.566750737 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:50.949040 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:50.949014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:51.013843 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.013785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:51.176999 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.176968 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t"] Oct 31 14:45:51.195010 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.194986 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" Oct 31 14:45:51.236460 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.236388 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t"] Oct 31 14:45:51.246683 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.246655 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:51.246837 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.246773 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nkdvg\"" Oct 31 14:45:51.246837 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.246778 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 14:45:51.331273 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.331244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slgb2\" (UniqueName: \"kubernetes.io/projected/f5edd464-29e1-4f1f-9f66-039ca19ba500-kube-api-access-slgb2\") pod \"migrator-66df5d496b-87j5t\" (UID: \"f5edd464-29e1-4f1f-9f66-039ca19ba500\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" Oct 31 14:45:51.347689 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.347649 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g9jmb"] Oct 31 14:45:51.352308 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:51.352281 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda21ed35_d5b8_4ef5_a6db_69b066ce2f6c.slice/crio-1640f95f05744c590925b22fecd3c17ccd8a54b474ec64ae87d69ea2d29befc9 WatchSource:0}: Error finding container 1640f95f05744c590925b22fecd3c17ccd8a54b474ec64ae87d69ea2d29befc9: Status 404 returned error can't find the container with id 1640f95f05744c590925b22fecd3c17ccd8a54b474ec64ae87d69ea2d29befc9 Oct 31 14:45:51.419050 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.419009 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fjcpl"] Oct 31 14:45:51.423307 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:51.423283 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8ff594_7d79_4f56_9eb4_a7af14808c03.slice/crio-6351fab7690f9518f1ea19685acea2d19432ed49132d6c2a8db4c7f530043a5d WatchSource:0}: Error finding container 6351fab7690f9518f1ea19685acea2d19432ed49132d6c2a8db4c7f530043a5d: Status 404 returned error can't find the container with id 6351fab7690f9518f1ea19685acea2d19432ed49132d6c2a8db4c7f530043a5d Oct 31 14:45:51.432489 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.432466 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slgb2\" (UniqueName: \"kubernetes.io/projected/f5edd464-29e1-4f1f-9f66-039ca19ba500-kube-api-access-slgb2\") pod \"migrator-66df5d496b-87j5t\" (UID: \"f5edd464-29e1-4f1f-9f66-039ca19ba500\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" Oct 31 14:45:51.489770 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.489716 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slgb2\" (UniqueName: \"kubernetes.io/projected/f5edd464-29e1-4f1f-9f66-039ca19ba500-kube-api-access-slgb2\") pod \"migrator-66df5d496b-87j5t\" (UID: \"f5edd464-29e1-4f1f-9f66-039ca19ba500\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" Oct 31 14:45:51.517674 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.517648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" Oct 31 14:45:51.634172 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.634143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:51.634325 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:51.634305 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:51.634381 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:51.634372 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls podName:6a4f5abc-05a0-4bef-ac48-b926c6b54a6a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:55.634358218 +0000 UTC m=+40.273876426 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls") pod "dns-default-nxdlw" (UID: "6a4f5abc-05a0-4bef-ac48-b926c6b54a6a") : secret "dns-default-metrics-tls" not found Oct 31 14:45:51.894830 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.894658 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t"] Oct 31 14:45:51.898867 ip-10-0-137-160 kubenswrapper[2578]: W1031 14:45:51.898843 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5edd464_29e1_4f1f_9f66_039ca19ba500.slice/crio-c27ba4a01873110a764e7a19d10324be9bc8b93d7dce69ef76c7e01d14563aac WatchSource:0}: Error finding container c27ba4a01873110a764e7a19d10324be9bc8b93d7dce69ef76c7e01d14563aac: Status 404 returned error can't find the container with id c27ba4a01873110a764e7a19d10324be9bc8b93d7dce69ef76c7e01d14563aac Oct 31 14:45:51.935511 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.935487 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:51.935601 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:51.935525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:51.935657 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:51.935610 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:51.935657 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:51.935637 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:51.935741 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:51.935668 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:53.93565263 +0000 UTC m=+38.575170843 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : configmap "kube-rbac-proxy" not found Oct 31 14:45:51.935741 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:51.935685 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:53.935676265 +0000 UTC m=+38.575194473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:52.080538 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:52.080506 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fjcpl" event={"ID":"8c8ff594-7d79-4f56-9eb4-a7af14808c03","Type":"ContainerStarted","Data":"6351fab7690f9518f1ea19685acea2d19432ed49132d6c2a8db4c7f530043a5d"} Oct 31 14:45:52.081670 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:52.081637 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" event={"ID":"f5edd464-29e1-4f1f-9f66-039ca19ba500","Type":"ContainerStarted","Data":"c27ba4a01873110a764e7a19d10324be9bc8b93d7dce69ef76c7e01d14563aac"} Oct 31 14:45:52.083169 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:52.083146 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wvbrg" event={"ID":"9ab9f675-4ee1-4226-a69d-2e362ccb753e","Type":"ContainerStarted","Data":"16a299cee4285e5922d6ea5d8e11349f13b5c8914acd096a1e7d0cee46485f68"} Oct 31 14:45:52.084307 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:52.084288 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g9jmb" event={"ID":"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c","Type":"ContainerStarted","Data":"1640f95f05744c590925b22fecd3c17ccd8a54b474ec64ae87d69ea2d29befc9"} Oct 31 14:45:52.197400 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:52.197355 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wvbrg" podStartSLOduration=1.9540770219999999 podStartE2EDuration="5.197341003s" totalImagesPullingTime="3.243263981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:52.196527555 +0000 UTC m=+36.836045784" watchObservedRunningTime="2026-10-31 14:45:52.197341003 +0000 UTC m=+36.836859232" Oct 31 14:45:53.953539 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:53.953505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:53.954051 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:53.953555 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:53.954051 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:53.953666 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:53.954051 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:53.953668 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:53.954051 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:53.953753 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.953734316 +0000 UTC m=+42.593252530 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:53.954051 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:53.953772 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:57.953764339 +0000 UTC m=+42.593282563 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : configmap "kube-rbac-proxy" not found Oct 31 14:45:55.095183 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:55.095147 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" event={"ID":"f5edd464-29e1-4f1f-9f66-039ca19ba500","Type":"ContainerStarted","Data":"5c1866d03fd4e84f9f0cc7ad0d1bb40cc7e6f5d9eddd45b1022e49e0bf1432e0"} Oct 31 14:45:55.216708 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:55.216644 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" podStartSLOduration=2.005095008 podStartE2EDuration="4.216627134s" totalImagesPullingTime="2.211532126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:55.215937366 +0000 UTC m=+39.855455591" watchObservedRunningTime="2026-10-31 14:45:55.216627134 +0000 UTC m=+39.856145366" Oct 31 14:45:55.667827 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:55.667796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:55.667960 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:55.667941 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:55.668024 ip-10-0-137-160 kubenswrapper[2578]: E1031 14:45:55.668011 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls podName:6a4f5abc-05a0-4bef-ac48-b926c6b54a6a nodeName:}" failed. No retries permitted until 2026-10-31 14:46:03.667995561 +0000 UTC m=+48.307513777 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls") pod "dns-default-nxdlw" (UID: "6a4f5abc-05a0-4bef-ac48-b926c6b54a6a") : secret "dns-default-metrics-tls" not found Oct 31 14:45:56.099718 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:56.099658 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g9jmb" event={"ID":"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c","Type":"ContainerStarted","Data":"d967c8eeb96716a0b50927d7e400eced01dc19c06abe679fa326c2f0b5438058"} Oct 31 14:45:56.213888 ip-10-0-137-160 kubenswrapper[2578]: I1031 14:45:56.213144 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g9jmb" podStartSLOduration=35.96591429 podStartE2EDuration="40.213129208s" totalImagesPullingTime="4.247214918s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:45:56.212564326 +0000 UTC m=+40.852082558" watchObservedRunningTime="2026-10-31 14:45:56.213129208 +0000 UTC m=+40.852647440" Oct 31 14:45:56.733156 ip-10-0-137-160 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 14:45:56.745088 ip-10-0-137-160 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 14:45:56.745258 ip-10-0-137-160 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 14:45:56.745305 ip-10-0-137-160 systemd[1]: kubelet.service: Consumed 2.309s CPU time. Oct 31 14:45:56.758269 ip-10-0-137-160 systemd[1]: Starting Kubernetes Kubelet... Oct 31 14:45:56.817068 ip-10-0-137-160 kubenswrapper[4399]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:56.817068 ip-10-0-137-160 kubenswrapper[4399]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 14:45:56.817068 ip-10-0-137-160 kubenswrapper[4399]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:56.817068 ip-10-0-137-160 kubenswrapper[4399]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 14:45:56.819180 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819115 4399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:56.819180 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819181 4399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819186 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819189 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819192 4399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819195 4399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819198 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819201 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819203 4399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819206 4399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819208 4399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819211 4399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819213 4399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819216 4399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819218 4399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819221 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819223 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819226 4399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819228 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819231 4399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819233 4399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:56.819252 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819236 4399 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819238 4399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819240 4399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819243 4399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819245 4399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819248 4399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819251 4399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819253 4399 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819261 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819265 4399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819267 4399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819270 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819272 4399 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819275 4399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819277 4399 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819280 4399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819282 4399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819285 4399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819287 4399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819290 4399 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:56.819766 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819292 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819295 4399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819297 4399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819300 4399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819302 4399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819304 4399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819307 4399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819309 4399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819311 4399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819319 4399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819322 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819325 4399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819327 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819331 4399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819335 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819338 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819341 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819344 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819347 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819350 4399 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:56.820256 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819352 4399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819355 4399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819357 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819360 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819362 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819364 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819367 4399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819369 4399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819372 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819374 4399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819376 4399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819379 4399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819382 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819384 4399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819386 4399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819389 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819391 4399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819394 4399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819396 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:56.820751 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819404 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819407 4399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819410 4399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819412 4399 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819416 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819418 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819421 4399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819423 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819426 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819428 4399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819431 4399 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819433 4399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819436 4399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819438 4399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819441 4399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819443 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819445 4399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819448 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819450 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:56.821216 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819453 4399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819455 4399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819458 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819460 4399 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819463 4399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819465 4399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.819468 4399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820209 4399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820216 4399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820219 4399 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820222 4399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820225 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820228 4399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820230 4399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820233 4399 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820236 4399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820239 4399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820242 4399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820244 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820247 4399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820249 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:56.821735 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820252 4399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820254 4399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820257 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820260 4399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820263 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820265 4399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820268 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820271 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820274 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820276 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820279 4399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820281 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820283 4399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820286 4399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820288 4399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820291 4399 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820293 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820295 4399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820298 4399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820300 4399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:56.822258 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820303 4399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820305 4399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820308 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820311 4399 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820313 4399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820316 4399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820319 4399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820322 4399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820324 4399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820327 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820329 4399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820331 4399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820335 4399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820337 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820339 4399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820342 4399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820344 4399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820347 4399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820349 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:56.822761 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820351 4399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820354 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820356 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820359 4399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820371 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820374 4399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820377 4399 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820379 4399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820382 4399 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820385 4399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820387 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820389 4399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820392 4399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820396 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820398 4399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820400 4399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820405 4399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820409 4399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820412 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:56.823249 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820414 4399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820418 4399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820420 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820423 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820425 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820428 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820430 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820433 4399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820435 4399 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820438 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820442 4399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820445 4399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820448 4399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820451 4399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820454 4399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820456 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820459 4399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820462 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820464 4399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:56.823732 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820467 4399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820469 4399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820472 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820474 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820477 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820479 4399 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820482 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820485 4399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820487 4399 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820491 4399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820493 4399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820496 4399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820498 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820501 4399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.820503 4399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821023 4399 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821029 4399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821031 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821034 4399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821036 4399 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:56.824181 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821039 4399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821042 4399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821045 4399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821047 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821056 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821059 4399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821062 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821065 4399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821067 4399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821070 4399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821072 4399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821075 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821077 4399 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821080 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821083 4399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821085 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821087 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821090 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821093 4399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:56.824679 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821095 4399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821098 4399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821100 4399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821103 4399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821106 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821108 4399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821110 4399 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821113 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821116 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821119 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821122 4399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821125 4399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821127 4399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821129 4399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821134 4399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821137 4399 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821140 4399 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821143 4399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821145 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:56.825166 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821148 4399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821150 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821153 4399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821155 4399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821158 4399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821160 4399 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821163 4399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821165 4399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821168 4399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821170 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821172 4399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821175 4399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821180 4399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821183 4399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821185 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821188 4399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821191 4399 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821193 4399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821196 4399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821198 4399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:56.825614 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821201 4399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821203 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821206 4399 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821209 4399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821211 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821214 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821216 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821219 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821222 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821224 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821227 4399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821229 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821232 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821235 4399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821237 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821239 4399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821242 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821244 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821247 4399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821249 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:56.826092 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821252 4399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821254 4399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821257 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821259 4399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821264 4399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821266 4399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821269 4399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821271 4399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821274 4399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821276 4399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821278 4399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821281 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821283 4399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821286 4399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821290 4399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821294 4399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821298 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821301 4399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821304 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:56.826560 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821306 4399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821308 4399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821311 4399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.821313 4399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821403 4399 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821422 4399 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821428 4399 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821433 4399 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821437 4399 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821440 4399 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821444 4399 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821448 4399 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821451 4399 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821454 4399 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821457 4399 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821460 4399 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821463 4399 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821467 4399 flags.go:64] FLAG: --cgroup-root="" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821470 4399 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821473 4399 flags.go:64] FLAG: --client-ca-file="" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821476 4399 flags.go:64] FLAG: --cloud-provider="external" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821479 4399 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821482 4399 flags.go:64] FLAG: --cluster-domain="" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821485 4399 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 14:45:56.827067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821488 4399 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821491 4399 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821494 4399 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821498 4399 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821500 4399 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821504 4399 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821507 4399 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821510 4399 flags.go:64] FLAG: --contention-profiling="false" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821513 4399 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821516 4399 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821519 4399 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821522 4399 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821526 4399 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821529 4399 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821532 4399 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821535 4399 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821538 4399 flags.go:64] FLAG: --enable-server="true" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821541 4399 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821545 4399 flags.go:64] FLAG: --event-burst="100" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821548 4399 flags.go:64] FLAG: --event-qps="50" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821551 4399 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821554 4399 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821557 4399 flags.go:64] FLAG: --eviction-hard="" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821561 4399 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 14:45:56.827624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821564 4399 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821568 4399 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821571 4399 flags.go:64] FLAG: --eviction-soft="" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821574 4399 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821577 4399 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821579 4399 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821582 4399 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821585 4399 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821588 4399 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821591 4399 flags.go:64] FLAG: --feature-gates="" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821596 4399 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821599 4399 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821602 4399 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821605 4399 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821608 4399 flags.go:64] FLAG: --healthz-port="10248" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821612 4399 flags.go:64] FLAG: --help="false" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821614 4399 flags.go:64] FLAG: --hostname-override="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821617 4399 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821620 4399 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821623 4399 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821626 4399 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821631 4399 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821633 4399 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 14:45:56.828243 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821636 4399 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821639 4399 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821642 4399 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821645 4399 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821648 4399 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821650 4399 flags.go:64] FLAG: --kube-reserved="" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821653 4399 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821656 4399 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821659 4399 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821661 4399 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821665 4399 flags.go:64] FLAG: --lock-file="" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821668 4399 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821671 4399 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821673 4399 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821683 4399 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821687 4399 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821690 4399 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821706 4399 flags.go:64] FLAG: --logging-format="text" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821711 4399 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821716 4399 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821721 4399 flags.go:64] FLAG: --manifest-url="" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821725 4399 flags.go:64] FLAG: --manifest-url-header="" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821732 4399 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821736 4399 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821747 4399 flags.go:64] FLAG: --max-pods="110" Oct 31 14:45:56.828812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821750 4399 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821753 4399 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821756 4399 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821759 4399 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821762 4399 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821765 4399 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821768 4399 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821775 4399 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821778 4399 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821781 4399 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821784 4399 flags.go:64] FLAG: --pod-cidr="" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821787 4399 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821789 4399 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821792 4399 flags.go:64] FLAG: --pods-per-core="0" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821795 4399 flags.go:64] FLAG: --port="10250" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821798 4399 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821801 4399 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0b98ce92b757cca08" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821805 4399 flags.go:64] FLAG: --qos-reserved="" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821808 4399 flags.go:64] FLAG: --read-only-port="10255" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821811 4399 flags.go:64] FLAG: --register-node="true" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821814 4399 flags.go:64] FLAG: --register-with-taints="" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821817 4399 flags.go:64] FLAG: --registry-burst="10" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821820 4399 flags.go:64] FLAG: --registry-qps="5" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821823 4399 flags.go:64] FLAG: --reserved-cpus="" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821825 4399 flags.go:64] FLAG: --reserved-memory="" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821829 4399 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 14:45:56.829568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821832 4399 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821835 4399 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821838 4399 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821841 4399 flags.go:64] FLAG: --runonce="false" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821843 4399 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821846 4399 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821850 4399 flags.go:64] FLAG: --seccomp-default="false" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821853 4399 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821856 4399 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821859 4399 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821862 4399 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821865 4399 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821867 4399 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821870 4399 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821873 4399 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821876 4399 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821879 4399 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821882 4399 flags.go:64] FLAG: --system-cgroups="" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821884 4399 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821889 4399 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821892 4399 flags.go:64] FLAG: --tls-cert-file="" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821895 4399 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821898 4399 flags.go:64] FLAG: --tls-min-version="" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821903 4399 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821906 4399 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 14:45:56.830370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821909 4399 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821911 4399 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821914 4399 flags.go:64] FLAG: --v="2" Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821919 4399 flags.go:64] FLAG: --version="false" Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821923 4399 flags.go:64] FLAG: --vmodule="" Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821926 4399 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.821929 4399 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822096 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822101 4399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822105 4399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822108 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822111 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822113 4399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822116 4399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822119 4399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822122 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822124 4399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822127 4399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822129 4399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822133 4399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:56.831425 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822137 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822140 4399 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822143 4399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822145 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822148 4399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822150 4399 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822153 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822155 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822158 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822160 4399 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822164 4399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822167 4399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822169 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822172 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822174 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822177 4399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822179 4399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822182 4399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822185 4399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822187 4399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:56.832097 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822190 4399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822192 4399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822194 4399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822197 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822199 4399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822202 4399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822205 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822207 4399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822210 4399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822212 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822215 4399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822218 4399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822220 4399 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822222 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822225 4399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822227 4399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822230 4399 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822232 4399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822235 4399 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822237 4399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:56.832813 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822239 4399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822242 4399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822246 4399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822248 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822251 4399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822253 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822256 4399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822258 4399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822261 4399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822264 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822266 4399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822269 4399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822272 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822274 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822278 4399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822280 4399 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822283 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822285 4399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822287 4399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:56.833449 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822290 4399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822293 4399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822295 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822298 4399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822300 4399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822303 4399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822305 4399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822307 4399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822310 4399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822313 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822315 4399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822317 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822320 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822322 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822325 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822328 4399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822331 4399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822333 4399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822336 4399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822339 4399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:56.833985 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822341 4399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822344 4399 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822346 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822349 4399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822351 4399 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822354 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822356 4399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822360 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822363 4399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822365 4399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822368 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822370 4399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822372 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822375 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.822401 4399 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:56.834534 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822439 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822443 4399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822446 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822449 4399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822451 4399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822454 4399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822456 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822459 4399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822461 4399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822464 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822467 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822469 4399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822472 4399 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822476 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822478 4399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822481 4399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822483 4399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822486 4399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822488 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:56.835019 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822490 4399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822493 4399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822495 4399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822498 4399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822500 4399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822504 4399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822506 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822509 4399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822511 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822514 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822516 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822519 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822521 4399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822524 4399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822527 4399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822529 4399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822532 4399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822534 4399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822537 4399 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822539 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:56.835486 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822542 4399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822544 4399 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822547 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822549 4399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822552 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822554 4399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822558 4399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822560 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822563 4399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822565 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822568 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822570 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822574 4399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822578 4399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822580 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822583 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822585 4399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822589 4399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822592 4399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:56.836124 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822594 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822597 4399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822599 4399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822602 4399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822607 4399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822610 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822613 4399 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822616 4399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822618 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822621 4399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822623 4399 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822625 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822628 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822631 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822634 4399 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822636 4399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822638 4399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822641 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822643 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822647 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:56.836689 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822649 4399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822652 4399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822654 4399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822657 4399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822659 4399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822662 4399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822665 4399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822667 4399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822670 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822672 4399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822676 4399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822679 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822682 4399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822684 4399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822686 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822689 4399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822692 4399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822708 4399 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822711 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822713 4399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:56.837234 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822716 4399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822719 4399 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822721 4399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822723 4399 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822726 4399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822729 4399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822732 4399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.822734 4399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.826851 4399 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.826863 4399 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.826880 4399 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.826885 4399 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826957 4399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826962 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826965 4399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826967 4399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826970 4399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826973 4399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826976 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:56.837817 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826978 4399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826980 4399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826983 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826986 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826988 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826990 4399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826993 4399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826995 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.826998 4399 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827000 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827003 4399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827006 4399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827008 4399 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827016 4399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827019 4399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827021 4399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827024 4399 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827026 4399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827029 4399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827031 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:56.838264 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827035 4399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827037 4399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827040 4399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827043 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827045 4399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827048 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827050 4399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827053 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827055 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827057 4399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827060 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827062 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827065 4399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827067 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827070 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827073 4399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827075 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827078 4399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827081 4399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827083 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:56.838848 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827086 4399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827088 4399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827091 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827093 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827095 4399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827098 4399 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827106 4399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827108 4399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827111 4399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827113 4399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827115 4399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827118 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827121 4399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827124 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827126 4399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827129 4399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827131 4399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827134 4399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827136 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:56.839363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827139 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827141 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827144 4399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827146 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827149 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827151 4399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827153 4399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827156 4399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827161 4399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827164 4399 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827168 4399 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827171 4399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827173 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827176 4399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827178 4399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827181 4399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827183 4399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827186 4399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827188 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827190 4399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:56.839871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827200 4399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827202 4399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827205 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827207 4399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827209 4399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827214 4399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827218 4399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827221 4399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827223 4399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827226 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827229 4399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827232 4399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827235 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827237 4399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827240 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827242 4399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827244 4399 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827247 4399 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827249 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:56.840340 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827251 4399 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.827280 4399 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827469 4399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827474 4399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827477 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827479 4399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827482 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827484 4399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827487 4399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827489 4399 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827491 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827494 4399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827496 4399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827499 4399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827507 4399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827510 4399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:56.840818 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827512 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827515 4399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827517 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827520 4399 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827522 4399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827525 4399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827527 4399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827530 4399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827532 4399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827535 4399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827544 4399 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827547 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827549 4399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827553 4399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827556 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827559 4399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827561 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827564 4399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827567 4399 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827569 4399 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:56.841190 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827572 4399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827575 4399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827579 4399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827581 4399 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827584 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827586 4399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827589 4399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827591 4399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827593 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827596 4399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827598 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827606 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827609 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827611 4399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827614 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827617 4399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827619 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827621 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827624 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:56.841680 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827626 4399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827629 4399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827631 4399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827634 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827636 4399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827639 4399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827641 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827643 4399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827646 4399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827648 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827650 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827653 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827655 4399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827658 4399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827660 4399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827663 4399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827665 4399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827667 4399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827670 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:56.842155 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827672 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827675 4399 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827677 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827680 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827682 4399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827685 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827712 4399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827717 4399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827721 4399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827724 4399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827727 4399 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827729 4399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827732 4399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827735 4399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827737 4399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827739 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827742 4399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827744 4399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827747 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827749 4399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:56.842609 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827752 4399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827754 4399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827756 4399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827759 4399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827761 4399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827764 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827766 4399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827769 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827771 4399 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827774 4399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827776 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827779 4399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827781 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827784 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.827809 4399 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 14:45:56.843103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827854 4399 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827858 4399 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827861 4399 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827864 4399 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827872 4399 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827875 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827878 4399 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827880 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827883 4399 feature_gate.go:386] unrecognized feature gate: Example Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827885 4399 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827887 4399 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827890 4399 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827893 4399 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827895 4399 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827897 4399 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827900 4399 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827902 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827906 4399 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827910 4399 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 14:45:56.843468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827914 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827917 4399 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827919 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827922 4399 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827924 4399 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827927 4399 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827929 4399 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827933 4399 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827936 4399 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827940 4399 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827943 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827946 4399 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827949 4399 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827951 4399 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827954 4399 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827956 4399 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827959 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827961 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827970 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827973 4399 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 14:45:56.843934 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827975 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827978 4399 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827980 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827983 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827985 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827988 4399 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827990 4399 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827992 4399 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827995 4399 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.827997 4399 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828001 4399 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828003 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828006 4399 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828008 4399 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828011 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828013 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828015 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828018 4399 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828020 4399 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 14:45:56.844472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828023 4399 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828025 4399 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828029 4399 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828032 4399 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828034 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828037 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828039 4399 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828042 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828044 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828046 4399 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828049 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828052 4399 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828060 4399 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828062 4399 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828065 4399 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828067 4399 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828070 4399 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828072 4399 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828075 4399 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828077 4399 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 14:45:56.844966 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828079 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828082 4399 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828085 4399 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828089 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828092 4399 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828094 4399 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828097 4399 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828099 4399 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828102 4399 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828104 4399 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828106 4399 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828109 4399 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828111 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828114 4399 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828118 4399 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828120 4399 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828123 4399 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828125 4399 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828128 4399 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 14:45:56.845725 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828130 4399 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828133 4399 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828135 4399 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828137 4399 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828140 4399 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828142 4399 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828150 4399 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828153 4399 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:45:56.828155 4399 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.828205 4399 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.829170 4399 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.829224 4399 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.829408 4399 server.go:1014] "Starting client certificate rotation" Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.829558 4399 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.829628 4399 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 14:40:16 +0000 UTC" deadline="2028-05-22 06:15:23.455468286 +0000 UTC" Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.829674 4399 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13647h29m26.625796549s" Oct 31 14:45:56.846505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.829942 4399 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 14:45:56.847166 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.832378 4399 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 14:45:56.847166 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.834138 4399 log.go:25] "Validated CRI v1 runtime API" Oct 31 14:45:56.847166 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.835780 4399 log.go:25] "Validated CRI v1 image API" Oct 31 14:45:56.847166 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.836594 4399 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 14:45:56.847166 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.838065 4399 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d8c27f2b-ce49-4d2b-9965-a422c97d2075:/dev/nvme0n1p3 e2843bd1-6e4b-45bc-aad3-42c48956d871:/dev/nvme0n1p4] Oct 31 14:45:56.847365 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.838085 4399 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e82938e30bb9ff73af930caac8ca143bc69b037e0bebfd2a7c7c6e820168d4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e82938e30bb9ff73af930caac8ca143bc69b037e0bebfd2a7c7c6e820168d4d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1640f95f05744c590925b22fecd3c17ccd8a54b474ec64ae87d69ea2d29befc9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1640f95f05744c590925b22fecd3c17ccd8a54b474ec64ae87d69ea2d29befc9/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18d33bcf35fdba83fdf3606b5c1b547b99f7b533c9a0782f36bbd42b2a73711f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18d33bcf35fdba83fdf3606b5c1b547b99f7b533c9a0782f36bbd42b2a73711f/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6351fab7690f9518f1ea19685acea2d19432ed49132d6c2a8db4c7f530043a5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6351fab7690f9518f1ea19685acea2d19432ed49132d6c2a8db4c7f530043a5d/userdata/shm major:0 minor:211 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e6096b961087455db523a013075d0205c051e01c9ca8d993bfa988000d8417d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e6096b961087455db523a013075d0205c051e01c9ca8d993bfa988000d8417d/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/874bf9688925c0c30d84e9717df36a5bb5a8d10fd9606c3afc8fd778b670cb5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/874bf9688925c0c30d84e9717df36a5bb5a8d10fd9606c3afc8fd778b670cb5a/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88b513be41a5c06f48052c4f5e97df1c7276061b7c4421e91c827a60fbfa6bb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88b513be41a5c06f48052c4f5e97df1c7276061b7c4421e91c827a60fbfa6bb6/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/890268a700baeec6115805f613da701bbfed12978caee4e3e0c261f3ab393c86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/890268a700baeec6115805f613da701bbfed12978caee4e3e0c261f3ab393c86/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8dd0f1b62270fd33b773b294481f95357d38f9d07a80c0157d3aaa6fb87bf512/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8dd0f1b62270fd33b773b294481f95357d38f9d07a80c0157d3aaa6fb87bf512/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/91e1571d431617f062f18653cd76d5a136a70ba466ed9c65873905303002ff06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/91e1571d431617f062f18653cd76d5a136a70ba466ed9c65873905303002ff06/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2dcbf4ff4819c9d38fda7b3267600e533cab5070b94bec430c7c78b813e7110/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2dcbf4ff4819c9d38fda7b3267600e533cab5070b94bec430c7c78b813e7110/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c27ba4a01873110a764e7a19d10324be9bc8b93d7dce69ef76c7e01d14563aac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c27ba4a01873110a764e7a19d10324be9bc8b93d7dce69ef76c7e01d14563aac/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de76c03d0de7ecbbd90d0570269358d81fc9f7b3e609924b2517643beaa13e86/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de76c03d0de7ecbbd90d0570269358d81fc9f7b3e609924b2517643beaa13e86/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb408cc802b20179cd8626d72b3cf38f69dc200174df8ffb547129e547fcf498/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb408cc802b20179cd8626d72b3cf38f69dc200174df8ffb547129e547fcf498/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0eeb3c8b-578d-4a72-af8f-9c8c53887516/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0eeb3c8b-578d-4a72-af8f-9c8c53887516/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14/volumes/kubernetes.io~projected/kube-api-access-fwktb:{mountpoint:/var/lib/kubelet/pods/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14/volumes/kubernetes.io~projected/kube-api-access-fwktb major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/254c6437-651d-484e-bbcc-6d6e393b88f7/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/254c6437-651d-484e-bbcc-6d6e393b88f7/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/254c6437-651d-484e-bbcc-6d6e393b88f7/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/254c6437-651d-484e-bbcc-6d6e393b88f7/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/254c6437-651d-484e-bbcc-6d6e393b88f7/volumes/kubernetes.io~projected/kube-api-access-mc2lq:{mountpoint:/var/lib/kubelet/pods/254c6437-651d-484e-bbcc-6d6e393b88f7/volumes/kubernetes.io~projected/kube-api-access-mc2lq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a602e7c-a12e-44d4-8dbe-a7a958e867b5/volumes/kubernetes.io~projected/kube-api-access-mgbc7:{mountpoint:/var/lib/kubelet/pods/5a602e7c-a12e-44d4-8dbe-a7a958e867b5/volumes/kubernetes.io~projected/kube-api-access-mgbc7 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5fabf907-4df5-4334-86e9-369dd561a536/volumes/kubernetes.io~projected/kube-api-access-rt6ss:{mountpoint:/var/lib/kubelet/pods/5fabf907-4df5-4334-86e9-369dd561a536/volumes/kubernetes.io~projected/kube-api-access-rt6ss major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a/volumes/kubernetes.io~projected/kube-api-access-5mdc9:{mountpoint:/var/lib/kubelet/pods/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a/volumes/kubernetes.io~projected/kube-api-access-5mdc9 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c8ff594-7d79-4f56-9eb4-a7af14808c03/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8c8ff594-7d79-4f56-9eb4-a7af14808c03/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ab9f675-4ee1-4226-a69d-2e362ccb753e/volumes/kubernetes.io~projected/kube-api-access-mqs5s:{mountpoint:/var/lib/kubelet/pods/9ab9f675-4ee1-4226-a69d-2e362ccb753e/volumes/kubernetes.io~projected/kube-api-access-mqs5s major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ab9f675-4ee1-4226-a69d-2e362ccb753e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/9ab9f675-4ee1-4226-a69d-2e362ccb753e/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba001443-fbac-4b4f-be07-179fcdefa740/volumes/kubernetes.io~projected/kube-api-access-8glxg:{mountpoint:/var/lib/kubelet/pods/ba001443-fbac-4b4f-be07-179fcdefa740/volumes/kubernetes.io~projected/kube-api-access-8glxg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf78675e-1580-42ee-a251-9f03b56179c7/volumes/kubernetes.io~projected/kube-api-access-cgrfh:{mountpoint:/var/lib/kubelet/pods/cf78675e-1580-42ee-a251-9f03b56179c7/volumes/kubernetes.io~projected/kube-api-access-cgrfh major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3bf4469-0f41-48b4-b3b5-e936ab2921ee/volumes/kubernetes.io~projected/kube-api-access-gb8tm:{mountpoint:/var/lib/kubelet/pods/d3bf4469-0f41-48b4-b3b5-e936ab2921ee/volumes/kubernetes.io~projected/kube-api-access-gb8tm major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c/volumes/kubernetes.io~projected/kube-api-access-txsxx:{mountpoint:/var/lib/kubelet/pods/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c/volumes/kubernetes.io~projected/kube-api-access-txsxx major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7903225-51db-41c7-a143-ea5b3b6a9fbe/volumes/kubernetes.io~projected/kube-api-access-92v4c:{mountpoint:/var/lib/kubelet/pods/e7903225-51db-41c7-a143-ea5b3b6a9fbe/volumes/kubernetes.io~projected/kube-api-access-92v4c major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a/volumes/kubernetes.io~projected/kube-api-access-4srfm:{mountpoint:/var/lib/kubelet/pods/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a/volumes/kubernetes.io~projected/kube-api-access-4srfm major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5edd464-29e1-4f1f-9f66-039ca19ba500/volumes/kubernetes.io~projected/kube-api-access-slgb2:{mountpoint:/var/lib/kubelet/pods/f5edd464-29e1-4f1f-9f66-039ca19ba500/volumes/kubernetes.io~projected/kube-api-access-slgb2 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f73cb9cb-94d1-453f-a199-3d8bd48a2079/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f73cb9cb-94d1-453f-a199-3d8bd48a2079/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f73cb9cb-94d1-453f-a199-3d8bd48a2079/volumes/kubernetes.io~projected/kube-api-access-jsjpl:{mountpoint:/var/lib/kubelet/pods/f73cb9cb-94d1-453f-a199-3d8bd48a2079/volumes/kubernetes.io~projected/kube-api-access-jsjpl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f73cb9cb-94d1-453f-a199-3d8bd48a2079/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f73cb9cb-94d1-453f-a199-3d8bd48a2079/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/847586ba5248876450d8fabdd1aaf1f15b7a3f0c12c017afacfaf4991d7138b6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/3006737d3b960cbf6cdd8fe32c1b16ecfbcbaccef2f6fb5fa87f9d95db512921/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/9a19087715f2f96311c583f8715139a21bfe1835b28f766f3d25253f5b4c55f0/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f051a3af50f81d33586d53d924143fb67e03c5b6f22812159964c1d5b26fbe15/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/089e2fa4ed5c7b5122c75629288df15653e81bb5d8281ae00d8d053b8ab89750/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/78bc063f34d177819712dc78c930ad60d588d5dee2ca1d16d901284cb20261dc/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/57fe6aa54611af5e10893e226193c7453fda04cbcbaeb7d8f5a97798c664d0a1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/a6d5356a46df13c6c09709bcd54228a858c20eefa7fecc553fb63a3bb313f4e8/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/560174976f3b5c1a72f15278cb8826bb54b5aff7b1d6d275e34614622ddc64a9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/1943fac2a298eabf932e91534255fc84fde7ba010ecf20b29dd44abc925b1c30/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/124815df1adb4fcdfa04a0a102a6b7cb35e3987cef382b5ff54a2efed5019aeb/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/2fe02fd8079e86a9df8f69d58165871aa7f0800ddbc26fd305fe53c670534b3f/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/231d1d2b787c006cbc4cf70f86ff4b6adedd1b42661754f4b22021564ebfc9ff/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5aeb4c4bc31e7f87b7fb8fb816b2911a7541e4470f404c7dc8815aa895018339/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d355b9df152ca99d3758ea6158df805b521d0ed769dfff4125c9d6f3a4155aab/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1e9c9c388d4ab7fbee52d490d2823179f1cfe64b58170a20aa5ca79e62b94c4a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/6ce89f68b28de2bb27a8d10ba7b1ac3c255e003ea61721b54c5426bf0100ebc6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/29f49ca76f3de56a5cebd6af897bbfd97d1a13407536aebeee2a8de91a6313f1/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/3a90c6cfaac389c94e80f16ccc32a88e3a98fad72b8459f181d1c13247d80854/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/96532457466efd170242be0d95d64c9eb3495a6b15f6adaf1feee46d6f8a4f80/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d1fdec5425486f89a521c48a86268cd22b8fb6f511bee08bd4710e81b4ecd713/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/a4e55da2f058755535a0de313c15ce29fc7a332b5844ca1d1a1799b7492b5d8a/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/35bebf1a91a5260ea6a4ba869901baf0fd68eefc259c4f30661cfba10ba41872/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/23777edf2cfe212d38794c19c67d645f6e48c78312a64fda83626285e46fe281/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/56a3790e56acbdabe5a0d6b0de040c98783e8e657b65621b837a3b6b93eebbd8/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/856b26ebf3f33003edf7502719fb1e3f53c17ac52e7509261f8dd4d245f6b4d6/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/88fbf2fb521724ddd03e8eb8cf6b4d3df449ba9514e2eac6347ec652a7ba1d6e/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/335a7f32fefb1e661aa8c20c3d84adf93e150f7f619e28c64cfd22f3100ced00/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/66621c275344357249f7f041ed430bb7cdf800ffcf0537f4f1bd7a01c26e9873/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/5939b9795b335473f828962e900eee808819a7b696e14fcbd868e5c53f65020b/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/23243262b7fa0eaaa433d30eee1f366403e0c5d8b33359314232429c62bc7676/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/88027f8d9d7d85828c3d1317b34d2da461f0cec4d2d3af2e9b4730d716b451d6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/fef0a52db7fdbdf8ad35ff142398e5492e8bf79de2d4784015b828e93f600f20/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/0978c9c1d92c0f9208ae2b9794280d695f40e657808de4cf6f0da385a13df36b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a35f76a5a75b68379b2eec8ccd93bca9f61eb02bbb98ff559161aeefde904aea/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/60e56ebc60e98938f8b76393ff497fc3cfd1b768f7cd732073588b494264efae/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/57eeaef6e765f3a2acb47fbaa6cf1e35cd3b58e2c82f0e7ed675b486d7c45a9e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/29508fc1414b418cae7886e44e3901f72460ff2fdbc06cb17e01ae1be9f4aa3a/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/5dc7d6ebfb1eb611151d35c641d20b4673c728eca2442d53f8fd54f865b4fc29/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.843934 4399 manager.go:217] Machine: {Timestamp:2026-10-31 14:45:56.843462031 +0000 UTC m=+0.054703434 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103925 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25cc9045f0cd0e61b04ba0bc4de700 SystemUUID:ec25cc90-45f0-cd0e-61b0-4ba0bc4de700 BootID:aee4a6c9-0428-4584-aea0-dc95f9cb6a86 Filesystems:[{Device:/var/lib/kubelet/pods/ba001443-fbac-4b4f-be07-179fcdefa740/volumes/kubernetes.io~projected/kube-api-access-8glxg DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c27ba4a01873110a764e7a19d10324be9bc8b93d7dce69ef76c7e01d14563aac/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a/volumes/kubernetes.io~projected/kube-api-access-4srfm DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c8ff594-7d79-4f56-9eb4-a7af14808c03/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f5edd464-29e1-4f1f-9f66-039ca19ba500/volumes/kubernetes.io~projected/kube-api-access-slgb2 DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/890268a700baeec6115805f613da701bbfed12978caee4e3e0c261f3ab393c86/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de76c03d0de7ecbbd90d0570269358d81fc9f7b3e609924b2517643beaa13e86/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb408cc802b20179cd8626d72b3cf38f69dc200174df8ffb547129e547fcf498/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/254c6437-651d-484e-bbcc-6d6e393b88f7/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e82938e30bb9ff73af930caac8ca143bc69b037e0bebfd2a7c7c6e820168d4d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ab9f675-4ee1-4226-a69d-2e362ccb753e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ab9f675-4ee1-4226-a69d-2e362ccb753e/volumes/kubernetes.io~projected/kube-api-access-mqs5s DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8dd0f1b62270fd33b773b294481f95357d38f9d07a80c0157d3aaa6fb87bf512/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/91e1571d431617f062f18653cd76d5a136a70ba466ed9c65873905303002ff06/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/874bf9688925c0c30d84e9717df36a5bb5a8d10fd9606c3afc8fd778b670cb5a/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0eeb3c8b-578d-4a72-af8f-9c8c53887516/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14/volumes/kubernetes.io~projected/kube-api-access-fwktb DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/5fabf907-4df5-4334-86e9-369dd561a536/volumes/kubernetes.io~projected/kube-api-access-rt6ss DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/cf78675e-1580-42ee-a251-9f03b56179c7/volumes/kubernetes.io~projected/kube-api-access-cgrfh DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5a602e7c-a12e-44d4-8dbe-a7a958e867b5/volumes/kubernetes.io~projected/kube-api-access-mgbc7 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e7903225-51db-41c7-a143-ea5b3b6a9fbe/volumes/kubernetes.io~projected/kube-api-access-92v4c DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f73cb9cb-94d1-453f-a199-3d8bd48a2079/volumes/kubernetes.io~projected/kube-api-access-jsjpl DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c/volumes/kubernetes.io~projected/kube-api-access-txsxx DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6351fab7690f9518f1ea19685acea2d19432ed49132d6c2a8db4c7f530043a5d/userdata/shm DeviceMajor:0 DeviceMinor:211 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2dcbf4ff4819c9d38fda7b3267600e533cab5070b94bec430c7c78b813e7110/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18d33bcf35fdba83fdf3606b5c1b547b99f7b533c9a0782f36bbd42b2a73711f/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a/volumes/kubernetes.io~projected/kube-api-access-5mdc9 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/254c6437-651d-484e-bbcc-6d6e393b88f7/volumes/kubernetes.io~projected/kube-api-access-mc2lq DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3bf4469-0f41-48b4-b3b5-e936ab2921ee/volumes/kubernetes.io~projected/kube-api-access-gb8tm DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88b513be41a5c06f48052c4f5e97df1c7276061b7c4421e91c827a60fbfa6bb6/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1640f95f05744c590925b22fecd3c17ccd8a54b474ec64ae87d69ea2d29befc9/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f73cb9cb-94d1-453f-a199-3d8bd48a2079/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/254c6437-651d-484e-bbcc-6d6e393b88f7/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e6096b961087455db523a013075d0205c051e01c9ca8d993bfa988000d8417d/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f73cb9cb-94d1-453f-a199-3d8bd48a2079/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1640f95f05744c5 MacAddress:72:ec:71:2c:79:5b Speed:10000 Mtu:8901} {Name:6351fab7690f951 MacAddress:9a:58:89:bc:1d:d7 Speed:10000 Mtu:8901} {Name:874bf9688925c0c MacAddress:56:74:a3:6f:15:cc Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:4f:44:c6:8d:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:4e:e3:70:33:a3:8c Speed:0 Mtu:8901} {Name:c27ba4a01873110 MacAddress:8e:fc:d2:9f:ec:f9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:4f:44:c6:8d:7d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:12:64:a6:2a:d4:2d Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:06:27:51:23:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844165 4399 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844208 4399 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844472 4399 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844491 4399 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-160.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844601 4399 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844609 4399 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844619 4399 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844632 4399 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844723 4399 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844868 4399 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844965 4399 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.844996 4399 kubelet.go:493] "Attempting to sync node with API server" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.845008 4399 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.845017 4399 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.845026 4399 kubelet.go:405] "Adding apiserver pod source" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.845036 4399 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.846946 4399 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.846963 4399 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 14:45:56.847532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.847508 4399 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 14:45:56.849596 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.849581 4399 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 14:45:56.849596 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.849597 4399 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 14:45:56.849720 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.849692 4399 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 14:45:56.849899 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.849889 4399 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 14:45:56.849932 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.849917 4399 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 14:45:56.850028 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850019 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 14:45:56.850060 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850034 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 14:45:56.850060 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850040 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 14:45:56.850060 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850047 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 14:45:56.850060 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850056 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 14:45:56.850156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850066 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 14:45:56.850156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850074 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 14:45:56.850156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850079 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 14:45:56.850156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850085 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 14:45:56.850156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850091 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 14:45:56.850156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850103 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 14:45:56.850156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850112 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 14:45:56.850156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850135 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 14:45:56.850156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.850145 4399 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 14:45:56.853362 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.853349 4399 server.go:1263] "Started kubelet" Oct 31 14:45:56.853457 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.853424 4399 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 14:45:56.853769 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.853444 4399 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 14:45:56.853769 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.853571 4399 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 14:45:56.853914 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.853881 4399 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 14:45:56.853858 ip-10-0-137-160 systemd[1]: Started Kubernetes Kubelet. Oct 31 14:45:56.854898 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.854744 4399 server.go:317] "Adding debug handlers to kubelet server" Oct 31 14:45:56.856209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.855965 4399 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 14:45:56.856209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.855990 4399 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 14:45:56.856209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.856080 4399 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 14:45:56.856209 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:56.856082 4399 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:56.856209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.856091 4399 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 14:45:56.856209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.856101 4399 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 14:45:56.856209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.856072 4399 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 14:40:18 +0000 UTC" deadline="2028-04-05 21:00:19.136478887 +0000 UTC" Oct 31 14:45:56.856209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.856123 4399 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12534h14m22.280359838s" Oct 31 14:45:56.857916 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.857796 4399 factory.go:55] Registering systemd factory Oct 31 14:45:56.857916 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.857819 4399 factory.go:223] Registration of the systemd container factory successfully Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.857912 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f73cb9cb-94d1-453f-a199-3d8bd48a2079" volumeName="kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-config" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.857949 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254c6437-651d-484e-bbcc-6d6e393b88f7" volumeName="kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-tmp" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.857962 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c8ff594-7d79-4f56-9eb4-a7af14808c03" volumeName="kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.857974 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eeb3c8b-578d-4a72-af8f-9c8c53887516" volumeName="kubernetes.io/secret/0eeb3c8b-578d-4a72-af8f-9c8c53887516-agent-certs" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.857999 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12f49f18-fdb2-4d17-ae2e-95ec4ea53b14" volumeName="kubernetes.io/projected/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-kube-api-access-fwktb" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858012 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a602e7c-a12e-44d4-8dbe-a7a958e867b5" volumeName="kubernetes.io/configmap/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-serviceca" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858022 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4f5abc-05a0-4bef-ac48-b926c6b54a6a" volumeName="kubernetes.io/empty-dir/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-tmp-dir" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858042 4399 factory.go:153] Registering CRI-O factory Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858059 4399 factory.go:223] Registration of the crio container factory successfully Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858099 4399 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858112 4399 factory.go:103] Registering Raw factory Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858122 4399 manager.go:1199] Started watching for new ooms in manager Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858046 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7903225-51db-41c7-a143-ea5b3b6a9fbe" volumeName="kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858170 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f73cb9cb-94d1-453f-a199-3d8bd48a2079" volumeName="kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-script-lib" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858190 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f73cb9cb-94d1-453f-a199-3d8bd48a2079" volumeName="kubernetes.io/secret/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 14:45:56.858196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858204 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba001443-fbac-4b4f-be07-179fcdefa740" volumeName="kubernetes.io/projected/ba001443-fbac-4b4f-be07-179fcdefa740-kube-api-access-8glxg" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858216 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7903225-51db-41c7-a143-ea5b3b6a9fbe" volumeName="kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858229 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7903225-51db-41c7-a143-ea5b3b6a9fbe" volumeName="kubernetes.io/projected/e7903225-51db-41c7-a143-ea5b3b6a9fbe-kube-api-access-92v4c" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858244 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0eeb3c8b-578d-4a72-af8f-9c8c53887516" volumeName="kubernetes.io/configmap/0eeb3c8b-578d-4a72-af8f-9c8c53887516-konnectivity-ca" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858284 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12f49f18-fdb2-4d17-ae2e-95ec4ea53b14" volumeName="kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cni-binary-copy" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858298 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a602e7c-a12e-44d4-8dbe-a7a958e867b5" volumeName="kubernetes.io/projected/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-kube-api-access-mgbc7" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858310 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ab9f675-4ee1-4226-a69d-2e362ccb753e" volumeName="kubernetes.io/projected/9ab9f675-4ee1-4226-a69d-2e362ccb753e-kube-api-access-mqs5s" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858322 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf78675e-1580-42ee-a251-9f03b56179c7" volumeName="kubernetes.io/configmap/cf78675e-1580-42ee-a251-9f03b56179c7-iptables-alerter-script" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858343 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf78675e-1580-42ee-a251-9f03b56179c7" volumeName="kubernetes.io/projected/cf78675e-1580-42ee-a251-9f03b56179c7-kube-api-access-cgrfh" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858358 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da21ed35-d5b8-4ef5-a6db-69b066ce2f6c" volumeName="kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858381 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4f5abc-05a0-4bef-ac48-b926c6b54a6a" volumeName="kubernetes.io/configmap/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-config-volume" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858410 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a4f5abc-05a0-4bef-ac48-b926c6b54a6a" volumeName="kubernetes.io/projected/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-kube-api-access-5mdc9" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858423 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ab9f675-4ee1-4226-a69d-2e362ccb753e" volumeName="kubernetes.io/secret/9ab9f675-4ee1-4226-a69d-2e362ccb753e-cert" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858435 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba001443-fbac-4b4f-be07-179fcdefa740" volumeName="kubernetes.io/empty-dir/ba001443-fbac-4b4f-be07-179fcdefa740-tmp-dir" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858455 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3bf4469-0f41-48b4-b3b5-e936ab2921ee" volumeName="kubernetes.io/empty-dir/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-data-volume" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858477 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f73cb9cb-94d1-453f-a199-3d8bd48a2079" volumeName="kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-env-overrides" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858499 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f73cb9cb-94d1-453f-a199-3d8bd48a2079" volumeName="kubernetes.io/projected/f73cb9cb-94d1-453f-a199-3d8bd48a2079-kube-api-access-jsjpl" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858510 4399 manager.go:319] Starting recovery of all containers Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858511 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12f49f18-fdb2-4d17-ae2e-95ec4ea53b14" volumeName="kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-daemon-config" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858562 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5edd464-29e1-4f1f-9f66-039ca19ba500" volumeName="kubernetes.io/projected/f5edd464-29e1-4f1f-9f66-039ca19ba500-kube-api-access-slgb2" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858580 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254c6437-651d-484e-bbcc-6d6e393b88f7" volumeName="kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-tuned" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858615 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fabf907-4df5-4334-86e9-369dd561a536" volumeName="kubernetes.io/projected/5fabf907-4df5-4334-86e9-369dd561a536-kube-api-access-rt6ss" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858628 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7903225-51db-41c7-a143-ea5b3b6a9fbe" volumeName="kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-binary-copy" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858650 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a" volumeName="kubernetes.io/projected/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-kube-api-access-4srfm" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858674 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="254c6437-651d-484e-bbcc-6d6e393b88f7" volumeName="kubernetes.io/projected/254c6437-651d-484e-bbcc-6d6e393b88f7-kube-api-access-mc2lq" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858720 4399 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3bf4469-0f41-48b4-b3b5-e936ab2921ee" volumeName="kubernetes.io/projected/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-api-access-gb8tm" seLinuxMountContext="" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858730 4399 reconstruct.go:97] "Volume reconstruction finished" Oct 31 14:45:56.858920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.858736 4399 reconciler.go:29] "Reconciler: start to sync state" Oct 31 14:45:56.887157 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.887125 4399 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 14:45:56.888502 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.888481 4399 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 14:45:56.888502 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.888506 4399 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 14:45:56.888664 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.888527 4399 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 14:45:56.888664 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:56.888567 4399 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 14:45:56.905668 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.905636 4399 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="4804601d61918dacda801090bb8483ed4ce3e248926fc721bfa8a1b2eb2b2e32" exitCode=0 Oct 31 14:45:56.905668 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.905668 4399 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="ac273263d9c1230c7f516193f8784248c26ace657dd1dc5f64d92a5e8c283708" exitCode=0 Oct 31 14:45:56.905858 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.905679 4399 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="71b9dd5c5daa65949e4e600d869e0357082bf9751bac65ca74780db23f51bb25" exitCode=0 Oct 31 14:45:56.905858 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.905715 4399 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="186d3f3ace1ca444ff3eb348ef2e0d6d27f4a009fa4bee2e628819943ed3ae5e" exitCode=0 Oct 31 14:45:56.905858 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.905726 4399 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="eaaecf8c2d27a32f0bd14d61466bc3505f554ac11242f7add7d40ff555ad55f6" exitCode=0 Oct 31 14:45:56.905858 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.905735 4399 generic.go:356] "Generic (PLEG): container finished" podID="e7903225-51db-41c7-a143-ea5b3b6a9fbe" containerID="ca0aae7cecffe1f77ad9e3e6d9053f589c8d6e644adc8e5eb734541f1ce6ea63" exitCode=0 Oct 31 14:45:56.914749 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.914721 4399 generic.go:356] "Generic (PLEG): container finished" podID="3f83f99f29758da3777ab885397f404f" containerID="4c8ddabcd862341a9cf446f6e61062a43a3b7a27a146698c3f5f2a008046f96a" exitCode=0 Oct 31 14:45:56.924373 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.924347 4399 manager.go:324] Recovery completed Oct 31 14:45:56.937072 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.937058 4399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:56.938109 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.938095 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:56.938219 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.938123 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:56.938219 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.938133 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:56.939375 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939357 4399 cpu_manager.go:226] "Starting" policy="none" Oct 31 14:45:56.939375 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939371 4399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:56.939474 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939373 4399 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 14:45:56.939474 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939438 4399 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 14:45:56.939579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939550 4399 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 14:45:56.939579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939562 4399 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 14:45:56.939579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939576 4399 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 14:45:56.939686 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939582 4399 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 14:45:56.939686 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939587 4399 policy_none.go:50] "Start" Oct 31 14:45:56.939686 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939598 4399 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 14:45:56.939686 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939607 4399 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 14:45:56.939876 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939734 4399 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 14:45:56.939876 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939741 4399 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 14:45:56.939876 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.939746 4399 policy_none.go:44] "Start" Oct 31 14:45:56.940192 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.940178 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:56.940245 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.940204 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:56.940245 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.940213 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:56.943534 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.943510 4399 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 14:45:56.946001 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.945990 4399 manager.go:341] "Starting Device Plugin manager" Oct 31 14:45:56.946073 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:56.946018 4399 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 14:45:56.946073 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.946026 4399 server.go:87] "Starting device plugin registration server" Oct 31 14:45:56.946318 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.946304 4399 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 14:45:56.946364 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.946319 4399 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 14:45:56.946412 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.946394 4399 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 14:45:56.946526 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.946494 4399 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 14:45:56.946526 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.946534 4399 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 14:45:56.946526 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.946541 4399 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 14:45:56.946681 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.946577 4399 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T14:45:56.946524699Z","UUID":"c4fd85a9-b9c3-4fee-9532-98180e8d95ba","Handler":null,"Name":"","Endpoint":""} Oct 31 14:45:56.947581 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:56.947565 4399 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 14:45:56.947676 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:56.947601 4399 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" Oct 31 14:45:56.947835 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.947813 4399 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 14:45:56.947936 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.947841 4399 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 14:45:56.989806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.989313 4399 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal"] Oct 31 14:45:56.989883 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.989876 4399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:56.990655 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.990633 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:56.990752 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.990662 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:56.990752 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.990674 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:56.990922 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.990911 4399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:56.991057 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991046 4399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:56.991604 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991584 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:56.991683 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991612 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:56.991683 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991628 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:56.991683 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991630 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:56.991683 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991648 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:56.991683 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991659 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:56.991856 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:56.991787 4399 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:56.991934 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991908 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"3f83f99f29758da3777ab885397f404f","Type":"ContainerStarted","Data":"f198c843599aa8213bc48ba07657e9dd244053fd10032c36643b966f9ae2bda6"} Oct 31 14:45:56.991963 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991940 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"3f83f99f29758da3777ab885397f404f","Type":"ContainerDied","Data":"4c8ddabcd862341a9cf446f6e61062a43a3b7a27a146698c3f5f2a008046f96a"} Oct 31 14:45:56.991963 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991954 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"3f83f99f29758da3777ab885397f404f","Type":"ContainerStarted","Data":"890268a700baeec6115805f613da701bbfed12978caee4e3e0c261f3ab393c86"} Oct 31 14:45:56.992018 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991965 4399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:56.992018 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991974 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"dc41cca5aaa38cfa8c691e2d5baf2778","Type":"ContainerStarted","Data":"5d8421d1c7db655639c0d72a3d183f03803926060f363bf56b53d22c03eb0507"} Oct 31 14:45:56.992018 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.991986 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" event={"ID":"dc41cca5aaa38cfa8c691e2d5baf2778","Type":"ContainerStarted","Data":"a2dcbf4ff4819c9d38fda7b3267600e533cab5070b94bec430c7c78b813e7110"} Oct 31 14:45:56.992624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.992611 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:56.992687 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.992631 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:56.992687 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:56.992639 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:56.992776 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:56.992765 4399 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-160.us-west-1.compute.internal\" not found" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.046468 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.046412 4399 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 14:45:57.047179 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.047166 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientMemory" Oct 31 14:45:57.047233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.047190 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Oct 31 14:45:57.047233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.047200 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeHasSufficientPID" Oct 31 14:45:57.047233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.047218 4399 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.055931 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.055897 4399 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:57.056053 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.056040 4399 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 14:45:57.105574 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.105545 4399 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:57.105668 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.105609 4399 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:57.105779 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.105764 4399 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 14:45:57.156568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.156551 4399 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.156633 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.156579 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.156633 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.156600 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc41cca5aaa38cfa8c691e2d5baf2778-config\") pod \"kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"dc41cca5aaa38cfa8c691e2d5baf2778\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.156633 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.156617 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.158441 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.158428 4399 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.158491 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.158480 4399 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.158953 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.158941 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeNotReady" Oct 31 14:45:57.159002 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.158968 4399 setters.go:546] "Node became not ready" node="ip-10-0-137-160.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T14:45:57Z","lastTransitionTime":"2026-10-31T14:45:57Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 31 14:45:57.203292 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.203271 4399 nodeinfomanager.go:471] existing CSINode "ip-10-0-137-160.us-west-1.compute.internal" is owned by different node (oldNodeID="3e1f2d48-e1a9-4daf-bf3d-1150dc05b779", newNodeID=""), cleaning up... Oct 31 14:45:57.215157 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.215138 4399 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:57.215212 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:57.215179 4399 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.215252 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.215214 4399 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.257333 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.257311 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.257414 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.257339 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.257414 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.257358 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc41cca5aaa38cfa8c691e2d5baf2778-config\") pod \"kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"dc41cca5aaa38cfa8c691e2d5baf2778\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.257414 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.257406 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc41cca5aaa38cfa8c691e2d5baf2778-config\") pod \"kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"dc41cca5aaa38cfa8c691e2d5baf2778\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.257505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.257412 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.257505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.257432 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3f83f99f29758da3777ab885397f404f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" (UID: \"3f83f99f29758da3777ab885397f404f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.272073 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.272054 4399 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:57.272117 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:57.272086 4399 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.845584 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.845556 4399 apiserver.go:52] "Watching apiserver" Oct 31 14:45:57.922873 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.922852 4399 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.923014 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.922998 4399 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.948179 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.948159 4399 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 14:45:57.948615 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.948592 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-nrbnb","kube-system/global-pull-secret-syncer-fjcpl","kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal","openshift-dns/node-resolver-rqlqd","openshift-multus/multus-t8nzm","openshift-insights/insights-runtime-extractor-zlmzj","openshift-ovn-kubernetes/ovnkube-node-7bxzn","openshift-cluster-node-tuning-operator/tuned-wp5zq","openshift-dns/dns-default-nxdlw","openshift-ingress-canary/ingress-canary-wvbrg","openshift-image-registry/node-ca-r2dg7","openshift-multus/network-metrics-daemon-tl4sm","openshift-network-operator/iptables-alerter-gmn9f","openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t","kube-system/konnectivity-agent-tqg7k","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp","openshift-network-diagnostics/network-check-target-g9jmb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal"] Oct 31 14:45:57.956034 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.956003 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:57.956809 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.956793 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:57.957434 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.957416 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:57.960112 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960093 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-var-lib-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.960197 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960118 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.960197 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960139 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8glxg\" (UniqueName: \"kubernetes.io/projected/ba001443-fbac-4b4f-be07-179fcdefa740-kube-api-access-8glxg\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:57.960197 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960155 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-conf\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.960197 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960170 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-device-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:57.960312 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960216 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdc9\" (UniqueName: \"kubernetes.io/projected/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-kube-api-access-5mdc9\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:57.960312 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960249 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-kubelet\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.960312 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960276 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.960312 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960297 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-config\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.960436 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960312 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovn-node-metrics-cert\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.960436 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960331 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-kubernetes\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.960436 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960391 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cnibin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.960532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960481 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-etc-kubernetes\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.960532 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960517 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba001443-fbac-4b4f-be07-179fcdefa740-hosts-file\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:57.960612 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960537 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-system-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.960612 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960551 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-multus\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.960612 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960582 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:57.960612 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960601 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:57.960798 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960632 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab9f675-4ee1-4226-a69d-2e362ccb753e-cert\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:57.960798 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960647 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-serviceca\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:57.960798 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960720 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbc7\" (UniqueName: \"kubernetes.io/projected/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-kube-api-access-mgbc7\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:57.960798 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960791 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgrfh\" (UniqueName: \"kubernetes.io/projected/cf78675e-1580-42ee-a251-9f03b56179c7-kube-api-access-cgrfh\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:57.960974 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960820 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-conf-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.960974 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960847 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:57.960974 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960874 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt6ss\" (UniqueName: \"kubernetes.io/projected/5fabf907-4df5-4334-86e9-369dd561a536-kube-api-access-rt6ss\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:57.960974 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960968 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-ovn\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.961122 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.960991 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-run\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961122 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961023 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-dbus\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:57.961122 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961046 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqs5s\" (UniqueName: \"kubernetes.io/projected/9ab9f675-4ee1-4226-a69d-2e362ccb753e-kube-api-access-mqs5s\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:57.961122 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961062 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-tmp-dir\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:57.961122 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961089 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-systemd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.961122 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961107 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.961321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961125 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-multus-certs\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.961321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961142 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srfm\" (UniqueName: \"kubernetes.io/projected/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-kube-api-access-4srfm\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:57.961321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961159 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cni-binary-copy\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.961321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961186 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-modprobe-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961237 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-tmp-dir\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:57.961321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961246 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-sys\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961293 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-tuned\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961558 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961349 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0eeb3c8b-578d-4a72-af8f-9c8c53887516-konnectivity-ca\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:57.961558 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961370 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-tuned\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961558 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961408 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-lib-modules\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961558 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961462 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-tmp\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961558 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961505 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-hostroot\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.961558 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961546 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/254c6437-651d-484e-bbcc-6d6e393b88f7-tmp\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961572 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-host\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:57.961806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961605 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-slash\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.961806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961634 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-env-overrides\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.961806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961660 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf78675e-1580-42ee-a251-9f03b56179c7-host-slash\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:57.961806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961710 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961732 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-systemd\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.961806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961752 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-bin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.961806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961770 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-node-log\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.961806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961801 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-log-socket\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961827 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-bin\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961852 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-netd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961872 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-netns\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961898 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwktb\" (UniqueName: \"kubernetes.io/projected/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-kube-api-access-fwktb\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.961923 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962031 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjpl\" (UniqueName: \"kubernetes.io/projected/f73cb9cb-94d1-453f-a199-3d8bd48a2079-kube-api-access-jsjpl\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962073 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0eeb3c8b-578d-4a72-af8f-9c8c53887516-agent-certs\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962107 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-kubelet-config\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962133 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962188 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962207 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-etc-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.962237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962230 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-host\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962269 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-socket-dir-parent\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962285 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-system-cni-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962300 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cnibin\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962317 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962346 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92v4c\" (UniqueName: \"kubernetes.io/projected/e7903225-51db-41c7-a143-ea5b3b6a9fbe-kube-api-access-92v4c\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962436 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962457 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-kubelet\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962474 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962497 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962526 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-k8s-cni-cncf-io\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962543 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-systemd-units\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962562 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-netns\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.962611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962579 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba001443-fbac-4b4f-be07-179fcdefa740-tmp-dir\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962649 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba001443-fbac-4b4f-be07-179fcdefa740-tmp-dir\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962670 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysconfig\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962759 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-var-lib-kubelet\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962777 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc2lq\" (UniqueName: \"kubernetes.io/projected/254c6437-651d-484e-bbcc-6d6e393b88f7-kube-api-access-mc2lq\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962817 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-script-lib\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962851 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-config-volume\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962888 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf78675e-1580-42ee-a251-9f03b56179c7-iptables-alerter-script\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962915 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-daemon-config\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962942 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962968 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.962995 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:57.963145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.963031 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:57.983580 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.983562 4399 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:57.983633 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:57.983597 4399 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:57.984418 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:57.984405 4399 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 14:45:57.984453 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:57.984437 4399 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-160.us-west-1.compute.internal" Oct 31 14:45:58.004446 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.004429 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\"" Oct 31 14:45:58.004715 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.004683 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 14:45:58.005028 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005008 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.005276 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005253 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 14:45:58.005386 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005334 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.005447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005416 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 14:45:58.005498 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005480 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.005498 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005480 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.005580 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005501 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 14:45:58.005580 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005517 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.005580 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005573 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\"" Oct 31 14:45:58.005675 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005611 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 14:45:58.005675 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005617 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 14:45:58.005675 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005617 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.005815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005680 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\"" Oct 31 14:45:58.005815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005728 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.005815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005757 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.005815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005762 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\"" Oct 31 14:45:58.005815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005796 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\"" Oct 31 14:45:58.005815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005807 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.005978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005827 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\"" Oct 31 14:45:58.005978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.005892 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.011119 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.011104 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-config\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.011163 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.011136 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-serviceca\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:58.011163 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.011141 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovn-node-metrics-cert\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.012236 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.012221 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-env-overrides\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.013463 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.013448 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f73cb9cb-94d1-453f-a199-3d8bd48a2079-ovnkube-script-lib\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.013596 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.013583 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cf78675e-1580-42ee-a251-9f03b56179c7-iptables-alerter-script\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:58.028027 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.028006 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 14:45:58.028027 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.028007 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 14:45:58.028170 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.028006 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 14:45:58.028250 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.028238 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 14:45:58.028310 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.028258 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\"" Oct 31 14:45:58.028466 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.028452 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\"" Oct 31 14:45:58.028527 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.028459 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 14:45:58.028527 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.028470 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nkdvg\"" Oct 31 14:45:58.029815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.029803 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\"" Oct 31 14:45:58.029815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.029808 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\"" Oct 31 14:45:58.030986 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.030972 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9ab9f675-4ee1-4226-a69d-2e362ccb753e-cert\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:58.034023 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.034009 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\"" Oct 31 14:45:58.034023 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.034017 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\"" Oct 31 14:45:58.034115 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.034028 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\"" Oct 31 14:45:58.034284 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.034274 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 14:45:58.034328 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.034288 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 14:45:58.034328 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.034309 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\"" Oct 31 14:45:58.042820 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.042805 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0eeb3c8b-578d-4a72-af8f-9c8c53887516-agent-certs\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:58.043937 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.043924 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c8ff594-7d79-4f56-9eb4-a7af14808c03-original-pull-secret\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:58.053184 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.053167 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\"" Oct 31 14:45:58.053738 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.053719 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgrfh\" (UniqueName: \"kubernetes.io/projected/cf78675e-1580-42ee-a251-9f03b56179c7-kube-api-access-cgrfh\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:58.053933 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.053914 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbc7\" (UniqueName: \"kubernetes.io/projected/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-kube-api-access-mgbc7\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:58.054020 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.054006 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjpl\" (UniqueName: \"kubernetes.io/projected/f73cb9cb-94d1-453f-a199-3d8bd48a2079-kube-api-access-jsjpl\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.054054 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.054011 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdc9\" (UniqueName: \"kubernetes.io/projected/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-kube-api-access-5mdc9\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:58.054054 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.054011 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8glxg\" (UniqueName: \"kubernetes.io/projected/ba001443-fbac-4b4f-be07-179fcdefa740-kube-api-access-8glxg\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:58.054186 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.054173 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txsxx\" (UniqueName: \"kubernetes.io/projected/da21ed35-d5b8-4ef5-a6db-69b066ce2f6c-kube-api-access-txsxx\") pod \"network-check-target-g9jmb\" (UID: \"da21ed35-d5b8-4ef5-a6db-69b066ce2f6c\") " pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:58.056681 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.056664 4399 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 14:45:58.064107 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064090 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-kubelet\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064186 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064113 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064186 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064130 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-kubernetes\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064186 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064145 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cnibin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064289 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064187 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064289 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064222 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-kubelet\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064289 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064229 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-kubernetes\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064289 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064252 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-etc-kubernetes\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064289 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064279 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba001443-fbac-4b4f-be07-179fcdefa740-hosts-file\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:58.064289 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064280 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-etc-kubernetes\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064300 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-system-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064314 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-multus\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064321 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ba001443-fbac-4b4f-be07-179fcdefa740-hosts-file\") pod \"node-resolver-rqlqd\" (UID: \"ba001443-fbac-4b4f-be07-179fcdefa740\") " pod="openshift-dns/node-resolver-rqlqd" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064298 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cnibin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064328 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064384 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-conf-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064384 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-multus\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064405 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064398 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-system-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064419 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-conf-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064422 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-ovn\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064401 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-kubelet-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064445 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-registration-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064446 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-run\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064468 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-dbus\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064467 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-run\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064448 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-ovn\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064502 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064520 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-dbus\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064548 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-systemd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064572 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064581 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-run-systemd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064604 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-multus-certs\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064616 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-cni-dir\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064628 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-multus-certs\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064642 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-modprobe-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064673 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-sys\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064757 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064771 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-modprobe-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064779 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-lib-modules\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064811 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-hostroot\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064826 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-host\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064840 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-data-volume\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064812 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-sys\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064855 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8tm\" (UniqueName: \"kubernetes.io/projected/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-api-access-gb8tm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064870 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-slash\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064884 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-hostroot\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064891 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a602e7c-a12e-44d4-8dbe-a7a958e867b5-host\") pod \"node-ca-r2dg7\" (UID: \"5a602e7c-a12e-44d4-8dbe-a7a958e867b5\") " pod="openshift-image-registry/node-ca-r2dg7" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064893 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf78675e-1580-42ee-a251-9f03b56179c7-host-slash\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064919 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-slash\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064930 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cf78675e-1580-42ee-a251-9f03b56179c7-host-slash\") pod \"iptables-alerter-gmn9f\" (UID: \"cf78675e-1580-42ee-a251-9f03b56179c7\") " pod="openshift-network-operator/iptables-alerter-gmn9f" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064880 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-lib-modules\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064943 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064948 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-data-volume\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.064994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.064971 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-systemd\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065012 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-bin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065026 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-node-log\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065059 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-node-log\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065082 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-systemd\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065092 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-cni-bin\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065111 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-log-socket\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065085 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-d\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065136 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-bin\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065150 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-log-socket\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065167 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-bin\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065171 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-netd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065196 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-netns\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065213 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065230 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-kubelet-config\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065241 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-cni-netd\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065243 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-netns\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065254 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065282 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065288 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-socket-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065294 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c8ff594-7d79-4f56-9eb4-a7af14808c03-kubelet-config\") pod \"global-pull-secret-syncer-fjcpl\" (UID: \"8c8ff594-7d79-4f56-9eb4-a7af14808c03\") " pod="kube-system/global-pull-secret-syncer-fjcpl" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065286 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-etc-selinux\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065307 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-etc-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065332 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-etc-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.065344 4399 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065344 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-host\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065372 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-socket-dir-parent\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065392 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-host\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.065397 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls podName:6a4f5abc-05a0-4bef-ac48-b926c6b54a6a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:58.565384516 +0000 UTC m=+1.776625937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls") pod "dns-default-nxdlw" (UID: "6a4f5abc-05a0-4bef-ac48-b926c6b54a6a") : secret "dns-default-metrics-tls" not found Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065410 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-system-cni-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065426 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cnibin\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065429 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-system-cni-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065425 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-socket-dir-parent\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065443 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065457 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cnibin\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065483 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065496 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-kubelet\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065530 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065533 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-var-lib-kubelet\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065564 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7903225-51db-41c7-a143-ea5b3b6a9fbe-tuning-conf-dir\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065593 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slgb2\" (UniqueName: \"kubernetes.io/projected/f5edd464-29e1-4f1f-9f66-039ca19ba500-kube-api-access-slgb2\") pod \"migrator-66df5d496b-87j5t\" (UID: \"f5edd464-29e1-4f1f-9f66-039ca19ba500\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065659 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-k8s-cni-cncf-io\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065676 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-systemd-units\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065692 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-netns\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065723 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysconfig\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065737 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-var-lib-kubelet\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065754 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065761 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-systemd-units\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065770 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065776 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-host-run-k8s-cni-cncf-io\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065780 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-run-netns\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065794 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-crio-socket\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065808 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysconfig\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065824 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-sys-fs\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065830 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-var-lib-kubelet\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065840 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-var-lib-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.065850 4399 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065862 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.065879 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:58.565870699 +0000 UTC m=+1.777112093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : secret "metrics-daemon-secret" not found Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065878 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-var-lib-openvswitch\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065893 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f73cb9cb-94d1-453f-a199-3d8bd48a2079-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7bxzn\" (UID: \"f73cb9cb-94d1-453f-a199-3d8bd48a2079\") " pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065898 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-conf\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065922 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-device-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065971 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5fabf907-4df5-4334-86e9-369dd561a536-device-dir\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.065978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.065983 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/254c6437-651d-484e-bbcc-6d6e393b88f7-etc-sysctl-conf\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.105316 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105253 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.105316 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105256 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.105316 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105278 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.105316 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105260 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.105316 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105297 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.105593 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105317 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.105593 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105331 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 14:45:58.105593 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105354 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 14:45:58.105593 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105549 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 14:45:58.105593 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105561 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.105593 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105574 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.105593 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105575 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.105832 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105600 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 14:45:58.105832 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105606 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.105832 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105607 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 14:45:58.105832 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105626 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 14:45:58.105832 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105631 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 14:45:58.105832 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105610 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 14:45:58.105832 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.105635 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 14:45:58.111743 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.111729 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.111784 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.111746 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0eeb3c8b-578d-4a72-af8f-9c8c53887516-konnectivity-ca\") pod \"konnectivity-agent-tqg7k\" (UID: \"0eeb3c8b-578d-4a72-af8f-9c8c53887516\") " pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:45:58.111813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.111794 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-cni-binary-copy\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.112855 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.112833 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-cni-binary-copy\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.114003 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.113977 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e7903225-51db-41c7-a143-ea5b3b6a9fbe-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.114094 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.114042 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-multus-daemon-config\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.114094 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.114046 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-config-volume\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:58.114368 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.114349 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srfm\" (UniqueName: \"kubernetes.io/projected/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-kube-api-access-4srfm\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:58.114492 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.114355 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwktb\" (UniqueName: \"kubernetes.io/projected/12f49f18-fdb2-4d17-ae2e-95ec4ea53b14-kube-api-access-fwktb\") pod \"multus-t8nzm\" (UID: \"12f49f18-fdb2-4d17-ae2e-95ec4ea53b14\") " pod="openshift-multus/multus-t8nzm" Oct 31 14:45:58.114492 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.114400 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92v4c\" (UniqueName: \"kubernetes.io/projected/e7903225-51db-41c7-a143-ea5b3b6a9fbe-kube-api-access-92v4c\") pod \"multus-additional-cni-plugins-nrbnb\" (UID: \"e7903225-51db-41c7-a143-ea5b3b6a9fbe\") " pod="openshift-multus/multus-additional-cni-plugins-nrbnb" Oct 31 14:45:58.114492 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.114453 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc2lq\" (UniqueName: \"kubernetes.io/projected/254c6437-651d-484e-bbcc-6d6e393b88f7-kube-api-access-mc2lq\") pod \"tuned-wp5zq\" (UID: \"254c6437-651d-484e-bbcc-6d6e393b88f7\") " pod="openshift-cluster-node-tuning-operator/tuned-wp5zq" Oct 31 14:45:58.114578 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.114515 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqs5s\" (UniqueName: \"kubernetes.io/projected/9ab9f675-4ee1-4226-a69d-2e362ccb753e-kube-api-access-mqs5s\") pod \"ingress-canary-wvbrg\" (UID: \"9ab9f675-4ee1-4226-a69d-2e362ccb753e\") " pod="openshift-ingress-canary/ingress-canary-wvbrg" Oct 31 14:45:58.114578 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.114562 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt6ss\" (UniqueName: \"kubernetes.io/projected/5fabf907-4df5-4334-86e9-369dd561a536-kube-api-access-rt6ss\") pod \"aws-ebs-csi-driver-node-cwkpp\" (UID: \"5fabf907-4df5-4334-86e9-369dd561a536\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-cwkpp" Oct 31 14:45:58.119433 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.119416 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8tm\" (UniqueName: \"kubernetes.io/projected/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-api-access-gb8tm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.119943 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.119930 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slgb2\" (UniqueName: \"kubernetes.io/projected/f5edd464-29e1-4f1f-9f66-039ca19ba500-kube-api-access-slgb2\") pod \"migrator-66df5d496b-87j5t\" (UID: \"f5edd464-29e1-4f1f-9f66-039ca19ba500\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-87j5t" Oct 31 14:45:58.166544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.166522 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.166610 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.166549 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.166610 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.166594 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-crio-socket\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.166682 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.166639 4399 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:58.166682 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.166649 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-crio-socket\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.166682 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.166651 4399 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:58.166783 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.166688 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:58.666676204 +0000 UTC m=+1.877917607 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : configmap "kube-rbac-proxy" not found Oct 31 14:45:58.166783 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.166725 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:58.666710214 +0000 UTC m=+1.877951619 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:58.569659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.569608 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:58.569859 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.569729 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:58.569859 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.569769 4399 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:45:58.569859 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.569818 4399 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:58.569859 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.569837 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.569823981 +0000 UTC m=+2.781065393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : secret "metrics-daemon-secret" not found Oct 31 14:45:58.570037 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.569892 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls podName:6a4f5abc-05a0-4bef-ac48-b926c6b54a6a nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.569881386 +0000 UTC m=+2.781122777 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls") pod "dns-default-nxdlw" (UID: "6a4f5abc-05a0-4bef-ac48-b926c6b54a6a") : secret "dns-default-metrics-tls" not found Oct 31 14:45:58.624958 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.624923 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:45:58.670270 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.670246 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.670375 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.670279 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:58.670454 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.670380 4399 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:58.670454 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.670421 4399 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:58.670536 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.670428 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.670415875 +0000 UTC m=+2.881657269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : configmap "kube-rbac-proxy" not found Oct 31 14:45:58.670536 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:58.670508 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:45:59.670491907 +0000 UTC m=+2.881733299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : secret "insights-runtime-extractor-tls" not found Oct 31 14:45:58.705624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.705599 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:45:58.707299 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:58.707280 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rqlqd_ba001443-fbac-4b4f-be07-179fcdefa740/dns-node-resolver/0.log" Oct 31 14:45:59.513718 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:59.513676 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-r2dg7_5a602e7c-a12e-44d4-8dbe-a7a958e867b5/node-ca/0.log" Oct 31 14:45:59.577136 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:59.577105 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:45:59.577267 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:59.577184 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:45:59.577304 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:59.577262 4399 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:45:59.577304 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:59.577290 4399 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:45:59.577370 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:59.577324 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:46:01.577307627 +0000 UTC m=+4.788549034 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : secret "metrics-daemon-secret" not found Oct 31 14:45:59.577370 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:59.577338 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls podName:6a4f5abc-05a0-4bef-ac48-b926c6b54a6a nodeName:}" failed. No retries permitted until 2026-10-31 14:46:01.577332046 +0000 UTC m=+4.788573436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls") pod "dns-default-nxdlw" (UID: "6a4f5abc-05a0-4bef-ac48-b926c6b54a6a") : secret "dns-default-metrics-tls" not found Oct 31 14:45:59.677732 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:59.677687 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:59.677837 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:45:59.677748 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:45:59.677895 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:59.677834 4399 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:45:59.677895 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:59.677841 4399 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:45:59.677895 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:59.677887 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:46:01.677875755 +0000 UTC m=+4.889117177 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : configmap "kube-rbac-proxy" not found Oct 31 14:45:59.677994 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:45:59.677899 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:46:01.677893221 +0000 UTC m=+4.889134611 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : secret "insights-runtime-extractor-tls" not found Oct 31 14:46:00.188403 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:00.188375 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:46:00.188642 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:00.188621 4399 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 14:46:00.507923 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:00.507884 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wvbrg_9ab9f675-4ee1-4226-a69d-2e362ccb753e/serve-healthcheck-canary/0.log" Oct 31 14:46:00.708524 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:00.708494 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-87j5t_f5edd464-29e1-4f1f-9f66-039ca19ba500/migrator/0.log" Oct 31 14:46:00.923853 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:00.923773 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tqg7k" Oct 31 14:46:01.610503 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:01.610472 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:46:01.610673 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:01.610511 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:46:01.610673 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:01.610615 4399 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 14:46:01.610673 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:01.610625 4399 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 14:46:01.610797 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:01.610685 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls podName:6a4f5abc-05a0-4bef-ac48-b926c6b54a6a nodeName:}" failed. No retries permitted until 2026-10-31 14:46:05.610671758 +0000 UTC m=+8.821913148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls") pod "dns-default-nxdlw" (UID: "6a4f5abc-05a0-4bef-ac48-b926c6b54a6a") : secret "dns-default-metrics-tls" not found Oct 31 14:46:01.610797 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:01.610721 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs podName:f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a nodeName:}" failed. No retries permitted until 2026-10-31 14:46:05.610710431 +0000 UTC m=+8.821951836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs") pod "network-metrics-daemon-tl4sm" (UID: "f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a") : secret "metrics-daemon-secret" not found Oct 31 14:46:01.711376 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:01.711350 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:46:01.711718 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:01.711381 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:46:01.711718 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:01.711454 4399 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 14:46:01.711718 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:01.711483 4399 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:46:01.711718 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:01.711497 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:46:05.711486468 +0000 UTC m=+8.922727859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : configmap "kube-rbac-proxy" not found Oct 31 14:46:01.711718 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:01.711541 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:46:05.71152952 +0000 UTC m=+8.922770942 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : secret "insights-runtime-extractor-tls" not found Oct 31 14:46:05.630659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.630628 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:46:05.631147 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.630674 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:46:05.631147 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.631043 4399 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 14:46:05.633300 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.633275 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/6a4f5abc-05a0-4bef-ac48-b926c6b54a6a-metrics-tls\") pod \"dns-default-nxdlw\" (UID: \"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a\") " pod="openshift-dns/dns-default-nxdlw" Oct 31 14:46:05.633409 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.633308 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a-metrics-certs\") pod \"network-metrics-daemon-tl4sm\" (UID: \"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a\") " pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:46:05.731482 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.731449 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:46:05.731482 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.731487 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:46:05.731623 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:05.731582 4399 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 14:46:05.731657 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:05.731644 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls podName:d3bf4469-0f41-48b4-b3b5-e936ab2921ee nodeName:}" failed. No retries permitted until 2026-10-31 14:46:13.73162847 +0000 UTC m=+16.942869863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zlmzj" (UID: "d3bf4469-0f41-48b4-b3b5-e936ab2921ee") : secret "insights-runtime-extractor-tls" not found Oct 31 14:46:05.731934 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.731918 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:46:05.757527 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.757508 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nxdlw" Oct 31 14:46:05.757611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:05.757593 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tl4sm" Oct 31 14:46:06.182864 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:06.182833 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tl4sm"] Oct 31 14:46:06.186976 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:06.186936 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fa7f29_6a1d_46e8_9cfa_e89a989bd63a.slice/crio-08fce4176add10d6e91b7a2b518d55613281b73c4024d4522ef413c90e212a86 WatchSource:0}: Error finding container 08fce4176add10d6e91b7a2b518d55613281b73c4024d4522ef413c90e212a86: Status 404 returned error can't find the container with id 08fce4176add10d6e91b7a2b518d55613281b73c4024d4522ef413c90e212a86 Oct 31 14:46:06.188864 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:06.188840 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:46:06.234027 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:06.234001 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nxdlw"] Oct 31 14:46:06.237641 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:06.237610 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a4f5abc_05a0_4bef_ac48_b926c6b54a6a.slice/crio-fe2cdaec4a7fad37c48a03a4a78bcb3056e6e29dad225c0d0a58eb4da96a494f WatchSource:0}: Error finding container fe2cdaec4a7fad37c48a03a4a78bcb3056e6e29dad225c0d0a58eb4da96a494f: Status 404 returned error can't find the container with id fe2cdaec4a7fad37c48a03a4a78bcb3056e6e29dad225c0d0a58eb4da96a494f Oct 31 14:46:06.942597 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:06.942558 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tl4sm" event={"ID":"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a","Type":"ContainerStarted","Data":"08fce4176add10d6e91b7a2b518d55613281b73c4024d4522ef413c90e212a86"} Oct 31 14:46:06.943549 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:06.943527 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxdlw" event={"ID":"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a","Type":"ContainerStarted","Data":"fe2cdaec4a7fad37c48a03a4a78bcb3056e6e29dad225c0d0a58eb4da96a494f"} Oct 31 14:46:07.616040 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:07.616005 4399 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-160.us-west-1.compute.internal" event="NodeReady" Oct 31 14:46:07.948642 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:07.948607 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tl4sm" event={"ID":"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a","Type":"ContainerStarted","Data":"7215ba4e84bd1f33946a796b179abf363bd345c7003a5be7ddc90e894c163b52"} Oct 31 14:46:07.949057 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:07.948672 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tl4sm" event={"ID":"f1fa7f29-6a1d-46e8-9cfa-e89a989bd63a","Type":"ContainerStarted","Data":"4c30940ed079204aa60fe798b554292ba75384742b720862e7235252257c53ac"} Oct 31 14:46:08.953581 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:08.953546 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxdlw" event={"ID":"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a","Type":"ContainerStarted","Data":"90a334246bcabf00074905274329380635efe30f99ee4d2f64e1480a2618360e"} Oct 31 14:46:08.954226 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:08.953597 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nxdlw" event={"ID":"6a4f5abc-05a0-4bef-ac48-b926c6b54a6a","Type":"ContainerStarted","Data":"96d567af4e3b093bc382210a94ce8a9997a838ce945cf2cacc5be05ed770bdf9"} Oct 31 14:46:08.954226 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:08.953772 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nxdlw" Oct 31 14:46:11.007046 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.007017 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-dgvjt"] Oct 31 14:46:11.008691 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.008675 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:11.034130 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.034109 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-dgvjt"] Oct 31 14:46:11.056554 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.056521 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-597c85d775-t24gx"] Oct 31 14:46:11.058214 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.058196 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.066809 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.066788 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2768\" (UniqueName: \"kubernetes.io/projected/4dbb3e48-f37e-43b4-a0af-af36b118b10e-kube-api-access-r2768\") pod \"downloads-6dd789c8b7-dgvjt\" (UID: \"4dbb3e48-f37e-43b4-a0af-af36b118b10e\") " pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:11.066902 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.066834 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4dbb3e48-f37e-43b4-a0af-af36b118b10e-tmp\") pod \"downloads-6dd789c8b7-dgvjt\" (UID: \"4dbb3e48-f37e-43b4-a0af-af36b118b10e\") " pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:11.082987 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.082957 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 14:46:11.083301 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.083277 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 14:46:11.083301 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.083286 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-6g8xr\"" Oct 31 14:46:11.112256 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.112232 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-597c85d775-t24gx"] Oct 31 14:46:11.167719 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.167676 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f497f3b9-e5ab-416d-a2ad-bf5123315e93-bound-sa-token\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.167823 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.167744 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2768\" (UniqueName: \"kubernetes.io/projected/4dbb3e48-f37e-43b4-a0af-af36b118b10e-kube-api-access-r2768\") pod \"downloads-6dd789c8b7-dgvjt\" (UID: \"4dbb3e48-f37e-43b4-a0af-af36b118b10e\") " pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:11.167823 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.167810 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f497f3b9-e5ab-416d-a2ad-bf5123315e93-image-registry-private-configuration\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.167932 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.167838 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f497f3b9-e5ab-416d-a2ad-bf5123315e93-registry-certificates\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.167932 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.167857 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f497f3b9-e5ab-416d-a2ad-bf5123315e93-installation-pull-secrets\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.167932 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.167875 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f497f3b9-e5ab-416d-a2ad-bf5123315e93-ca-trust-extracted\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.167932 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.167927 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f497f3b9-e5ab-416d-a2ad-bf5123315e93-registry-tls\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.168082 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.167962 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f497f3b9-e5ab-416d-a2ad-bf5123315e93-trusted-ca\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.168082 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.168001 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwm6\" (UniqueName: \"kubernetes.io/projected/f497f3b9-e5ab-416d-a2ad-bf5123315e93-kube-api-access-8dwm6\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.168082 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.168059 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4dbb3e48-f37e-43b4-a0af-af36b118b10e-tmp\") pod \"downloads-6dd789c8b7-dgvjt\" (UID: \"4dbb3e48-f37e-43b4-a0af-af36b118b10e\") " pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:11.168349 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.168334 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4dbb3e48-f37e-43b4-a0af-af36b118b10e-tmp\") pod \"downloads-6dd789c8b7-dgvjt\" (UID: \"4dbb3e48-f37e-43b4-a0af-af36b118b10e\") " pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:11.182323 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.182302 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 14:46:11.182520 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.182497 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 14:46:11.182673 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.182657 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 14:46:11.182813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.182801 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-89zxn\"" Oct 31 14:46:11.182849 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.182834 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 14:46:11.226759 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.226736 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2768\" (UniqueName: \"kubernetes.io/projected/4dbb3e48-f37e-43b4-a0af-af36b118b10e-kube-api-access-r2768\") pod \"downloads-6dd789c8b7-dgvjt\" (UID: \"4dbb3e48-f37e-43b4-a0af-af36b118b10e\") " pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:11.268552 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.268502 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f497f3b9-e5ab-416d-a2ad-bf5123315e93-image-registry-private-configuration\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.268552 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.268529 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f497f3b9-e5ab-416d-a2ad-bf5123315e93-registry-certificates\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.268552 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.268545 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f497f3b9-e5ab-416d-a2ad-bf5123315e93-installation-pull-secrets\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.268767 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.268569 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f497f3b9-e5ab-416d-a2ad-bf5123315e93-ca-trust-extracted\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.268767 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.268597 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f497f3b9-e5ab-416d-a2ad-bf5123315e93-registry-tls\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.268767 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.268618 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f497f3b9-e5ab-416d-a2ad-bf5123315e93-trusted-ca\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.268767 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.268647 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwm6\" (UniqueName: \"kubernetes.io/projected/f497f3b9-e5ab-416d-a2ad-bf5123315e93-kube-api-access-8dwm6\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.268767 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.268727 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f497f3b9-e5ab-416d-a2ad-bf5123315e93-bound-sa-token\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.269105 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.269049 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f497f3b9-e5ab-416d-a2ad-bf5123315e93-ca-trust-extracted\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.269469 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.269443 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f497f3b9-e5ab-416d-a2ad-bf5123315e93-registry-certificates\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.269812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.269792 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f497f3b9-e5ab-416d-a2ad-bf5123315e93-trusted-ca\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.271274 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.271255 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f497f3b9-e5ab-416d-a2ad-bf5123315e93-registry-tls\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.271583 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.271564 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f497f3b9-e5ab-416d-a2ad-bf5123315e93-image-registry-private-configuration\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.271639 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.271615 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f497f3b9-e5ab-416d-a2ad-bf5123315e93-installation-pull-secrets\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.317815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.317790 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:11.325980 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.325959 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f497f3b9-e5ab-416d-a2ad-bf5123315e93-bound-sa-token\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.327680 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.327661 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwm6\" (UniqueName: \"kubernetes.io/projected/f497f3b9-e5ab-416d-a2ad-bf5123315e93-kube-api-access-8dwm6\") pod \"image-registry-597c85d775-t24gx\" (UID: \"f497f3b9-e5ab-416d-a2ad-bf5123315e93\") " pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.367323 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.367292 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.738931 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.738893 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-597c85d775-t24gx"] Oct 31 14:46:11.741809 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.741786 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-dgvjt"] Oct 31 14:46:11.743310 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:11.743280 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf497f3b9_e5ab_416d_a2ad_bf5123315e93.slice/crio-6923fdb06de3569b2a325f4a900e610c6faee7eb119f6975a5cd02f2c6b6c746 WatchSource:0}: Error finding container 6923fdb06de3569b2a325f4a900e610c6faee7eb119f6975a5cd02f2c6b6c746: Status 404 returned error can't find the container with id 6923fdb06de3569b2a325f4a900e610c6faee7eb119f6975a5cd02f2c6b6c746 Oct 31 14:46:11.781066 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:11.781041 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbb3e48_f37e_43b4_a0af_af36b118b10e.slice/crio-edef582c333074d54468e2a27af352aab676485d83e3ea35360fd435bc883df2 WatchSource:0}: Error finding container edef582c333074d54468e2a27af352aab676485d83e3ea35360fd435bc883df2: Status 404 returned error can't find the container with id edef582c333074d54468e2a27af352aab676485d83e3ea35360fd435bc883df2 Oct 31 14:46:11.966753 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.966718 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-597c85d775-t24gx" event={"ID":"f497f3b9-e5ab-416d-a2ad-bf5123315e93","Type":"ContainerStarted","Data":"fa15aee0c6cad868b4ce978783b807b6ec41f7bec6aa3f6dcee9edcb497ca97f"} Oct 31 14:46:11.966930 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.966774 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-597c85d775-t24gx" event={"ID":"f497f3b9-e5ab-416d-a2ad-bf5123315e93","Type":"ContainerStarted","Data":"6923fdb06de3569b2a325f4a900e610c6faee7eb119f6975a5cd02f2c6b6c746"} Oct 31 14:46:11.966930 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.966867 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:11.967981 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:11.967952 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-dgvjt" event={"ID":"4dbb3e48-f37e-43b4-a0af-af36b118b10e","Type":"ContainerStarted","Data":"edef582c333074d54468e2a27af352aab676485d83e3ea35360fd435bc883df2"} Oct 31 14:46:12.084393 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:12.084100 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-597c85d775-t24gx" podStartSLOduration=1.084087115 podStartE2EDuration="1.084087115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:12.083661593 +0000 UTC m=+15.294903007" watchObservedRunningTime="2026-10-31 14:46:12.084087115 +0000 UTC m=+15.295328527" Oct 31 14:46:13.793278 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:13.793242 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:46:13.795887 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:13.795862 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d3bf4469-0f41-48b4-b3b5-e936ab2921ee-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zlmzj\" (UID: \"d3bf4469-0f41-48b4-b3b5-e936ab2921ee\") " pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:46:13.858104 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:13.858070 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zlmzj" Oct 31 14:46:14.235531 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:14.235497 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zlmzj"] Oct 31 14:46:14.240035 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:14.240008 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3bf4469_0f41_48b4_b3b5_e936ab2921ee.slice/crio-3b325d0f47f782b642825696ccde359dc8d4a92bdd2a8758dd8c7017a8e09be8 WatchSource:0}: Error finding container 3b325d0f47f782b642825696ccde359dc8d4a92bdd2a8758dd8c7017a8e09be8: Status 404 returned error can't find the container with id 3b325d0f47f782b642825696ccde359dc8d4a92bdd2a8758dd8c7017a8e09be8 Oct 31 14:46:14.981439 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:14.981391 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zlmzj" event={"ID":"d3bf4469-0f41-48b4-b3b5-e936ab2921ee","Type":"ContainerStarted","Data":"a72e8c5d5fe097322b575c1983278d4eaa1b3d192c4793413207b8e6ece87180"} Oct 31 14:46:14.981915 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:14.981453 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zlmzj" event={"ID":"d3bf4469-0f41-48b4-b3b5-e936ab2921ee","Type":"ContainerStarted","Data":"3b325d0f47f782b642825696ccde359dc8d4a92bdd2a8758dd8c7017a8e09be8"} Oct 31 14:46:15.987864 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:15.987826 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zlmzj" event={"ID":"d3bf4469-0f41-48b4-b3b5-e936ab2921ee","Type":"ContainerStarted","Data":"8eb2244615152b29fd3c139dade1af7647791ed94d034762c9f66b79cd37f9f5"} Oct 31 14:46:18.000048 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:18.000009 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zlmzj" event={"ID":"d3bf4469-0f41-48b4-b3b5-e936ab2921ee","Type":"ContainerStarted","Data":"c5a23755e13760056653b1f266163cbcc4f279f1e790bbe438f0cd21fc9b3045"} Oct 31 14:46:18.961877 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:18.961843 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nxdlw" Oct 31 14:46:19.912278 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:19.912250 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7bxzn" Oct 31 14:46:26.207632 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:26.207592 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:46:27.694418 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.694384 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7"] Oct 31 14:46:27.697191 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.697172 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.784766 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.784724 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fjl\" (UniqueName: \"kubernetes.io/projected/166f6dae-eda8-4568-8183-20db7b0dd0b6-kube-api-access-24fjl\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.784944 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.784835 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/166f6dae-eda8-4568-8183-20db7b0dd0b6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.784944 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.784872 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/166f6dae-eda8-4568-8183-20db7b0dd0b6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.785067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.784985 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/166f6dae-eda8-4568-8183-20db7b0dd0b6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.799225 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.799200 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 14:46:27.799225 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.799200 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 14:46:27.799360 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.799273 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 14:46:27.799884 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.799861 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7"] Oct 31 14:46:27.807606 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.807583 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k6kcz\"" Oct 31 14:46:27.850563 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.850529 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2jxj6"] Oct 31 14:46:27.851865 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.851843 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 14:46:27.851970 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.851866 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 14:46:27.852579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.852561 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:27.885570 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.885545 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/166f6dae-eda8-4568-8183-20db7b0dd0b6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.885722 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.885585 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fjl\" (UniqueName: \"kubernetes.io/projected/166f6dae-eda8-4568-8183-20db7b0dd0b6-kube-api-access-24fjl\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.885722 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.885618 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/166f6dae-eda8-4568-8183-20db7b0dd0b6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.885722 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.885640 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/166f6dae-eda8-4568-8183-20db7b0dd0b6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.886732 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.886691 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/166f6dae-eda8-4568-8183-20db7b0dd0b6-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.888464 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.888430 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/166f6dae-eda8-4568-8183-20db7b0dd0b6-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.888464 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.888449 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/166f6dae-eda8-4568-8183-20db7b0dd0b6-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.921191 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.921168 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 14:46:27.942714 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.942668 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fjl\" (UniqueName: \"kubernetes.io/projected/166f6dae-eda8-4568-8183-20db7b0dd0b6-kube-api-access-24fjl\") pod \"openshift-state-metrics-7bd7489957-gz5c7\" (UID: \"166f6dae-eda8-4568-8183-20db7b0dd0b6\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:27.986806 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.986775 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:27.986970 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.986816 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-metrics-client-ca\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:27.986970 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.986843 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-tls\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:27.986970 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.986915 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdlbk\" (UniqueName: \"kubernetes.io/projected/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-kube-api-access-gdlbk\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:27.987065 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.987007 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-textfile\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:27.987065 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.987033 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-wtmp\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:27.987065 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.987048 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-sys\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:27.987172 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.987072 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-root\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:27.987172 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:27.987109 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-accelerators-collector-config\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.001368 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.001349 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwnt5\"" Oct 31 14:46:28.001368 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.001364 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 14:46:28.001529 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.001445 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 14:46:28.010576 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.010556 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" Oct 31 14:46:28.088294 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088269 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-textfile\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.088406 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088319 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-wtmp\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.088406 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088350 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-sys\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.088406 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088400 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-root\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.088512 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088429 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-accelerators-collector-config\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.088512 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088480 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.088579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088519 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-metrics-client-ca\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.088579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088549 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-tls\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.088579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088570 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlbk\" (UniqueName: \"kubernetes.io/projected/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-kube-api-access-gdlbk\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.089754 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.088757 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-root\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.089754 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.089016 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-textfile\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.089754 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.089121 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-wtmp\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.089754 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.089149 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-sys\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.089754 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.089461 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-accelerators-collector-config\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.089754 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.089498 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-metrics-client-ca\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.092603 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.092328 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-tls\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.092603 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.092552 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.152321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.152295 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlbk\" (UniqueName: \"kubernetes.io/projected/f6e52f66-5e0e-4cbb-852d-fd4aa259d271-kube-api-access-gdlbk\") pod \"node-exporter-2jxj6\" (UID: \"f6e52f66-5e0e-4cbb-852d-fd4aa259d271\") " pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.166221 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.166199 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2jxj6" Oct 31 14:46:28.176299 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:28.176273 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e52f66_5e0e_4cbb_852d_fd4aa259d271.slice/crio-565184d12b832870a49628e0ae02e6374dbd66bb8f5489b64edaa50631a01518 WatchSource:0}: Error finding container 565184d12b832870a49628e0ae02e6374dbd66bb8f5489b64edaa50631a01518: Status 404 returned error can't find the container with id 565184d12b832870a49628e0ae02e6374dbd66bb8f5489b64edaa50631a01518 Oct 31 14:46:28.435230 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.435141 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7"] Oct 31 14:46:28.439829 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:28.439802 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod166f6dae_eda8_4568_8183_20db7b0dd0b6.slice/crio-e4822c51ab3c11ef41f27d37e4397b0cb93726900bc181b4a244efa8f91fcb5a WatchSource:0}: Error finding container e4822c51ab3c11ef41f27d37e4397b0cb93726900bc181b4a244efa8f91fcb5a: Status 404 returned error can't find the container with id e4822c51ab3c11ef41f27d37e4397b0cb93726900bc181b4a244efa8f91fcb5a Oct 31 14:46:28.931538 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:28.929416 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g9jmb" Oct 31 14:46:29.041231 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:29.041181 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2jxj6" event={"ID":"f6e52f66-5e0e-4cbb-852d-fd4aa259d271","Type":"ContainerStarted","Data":"565184d12b832870a49628e0ae02e6374dbd66bb8f5489b64edaa50631a01518"} Oct 31 14:46:29.043643 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:29.043608 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" event={"ID":"166f6dae-eda8-4568-8183-20db7b0dd0b6","Type":"ContainerStarted","Data":"24fd971bb8f71ee1c9e2f4e30769fce94c5d67df0f5aa27c058fd16d3e25739e"} Oct 31 14:46:29.043790 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:29.043671 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" event={"ID":"166f6dae-eda8-4568-8183-20db7b0dd0b6","Type":"ContainerStarted","Data":"739cfec6a9c099c49cb98a1c046d423553cf01c804794b74d33c752ee9aa92ba"} Oct 31 14:46:29.043790 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:29.043720 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" event={"ID":"166f6dae-eda8-4568-8183-20db7b0dd0b6","Type":"ContainerStarted","Data":"e4822c51ab3c11ef41f27d37e4397b0cb93726900bc181b4a244efa8f91fcb5a"} Oct 31 14:46:29.045561 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:29.045530 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-dgvjt" event={"ID":"4dbb3e48-f37e-43b4-a0af-af36b118b10e","Type":"ContainerStarted","Data":"c54a3df7ad26fd9802502e51baebc79897dc63d1b5f0ccc49f4815d280717e73"} Oct 31 14:46:29.045888 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:29.045859 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:29.062687 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:29.062663 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-dgvjt" Oct 31 14:46:29.164586 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:29.164530 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-dgvjt" podStartSLOduration=2.934888872 podStartE2EDuration="19.164511261s" totalImagesPullingTime="16.229622389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:29.163216353 +0000 UTC m=+32.374457769" watchObservedRunningTime="2026-10-31 14:46:29.164511261 +0000 UTC m=+32.375752674" Oct 31 14:46:30.497402 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.497367 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6646454b57-j46xn"] Oct 31 14:46:30.520138 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.520099 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6646454b57-j46xn"] Oct 31 14:46:30.520298 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.520239 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.572904 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.572878 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 14:46:30.572904 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.572901 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 14:46:30.573073 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.572881 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 14:46:30.573073 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.572961 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 14:46:30.573073 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.572988 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m4dnq\"" Oct 31 14:46:30.573073 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.573015 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 14:46:30.608728 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.608675 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-grpc-tls\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.608836 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.608788 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8008214d-c86a-4ff0-9838-54c39f5d2a81-tmp\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.608896 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.608872 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-tls\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.608953 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.608913 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8008214d-c86a-4ff0-9838-54c39f5d2a81-metrics-client-ca\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.609002 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.608978 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64k57\" (UniqueName: \"kubernetes.io/projected/8008214d-c86a-4ff0-9838-54c39f5d2a81-kube-api-access-64k57\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.609079 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.609056 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.609183 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.609159 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.609261 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.609200 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.609261 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.609239 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.673714 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.673670 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-78mefrvckg941\"" Oct 31 14:46:30.709764 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.709734 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64k57\" (UniqueName: \"kubernetes.io/projected/8008214d-c86a-4ff0-9838-54c39f5d2a81-kube-api-access-64k57\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.709899 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.709778 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.709899 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.709823 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.709899 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.709856 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.709899 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.709896 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.710089 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.709925 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-grpc-tls\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.710089 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.709975 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8008214d-c86a-4ff0-9838-54c39f5d2a81-tmp\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.710089 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.710026 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-tls\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.710089 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.710068 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8008214d-c86a-4ff0-9838-54c39f5d2a81-metrics-client-ca\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.711941 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.710958 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8008214d-c86a-4ff0-9838-54c39f5d2a81-metrics-client-ca\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.712096 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.710996 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8008214d-c86a-4ff0-9838-54c39f5d2a81-tmp\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.713151 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.713105 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.714246 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.714214 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-tls\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.714447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.714421 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-grpc-tls\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.715248 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.714843 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.715248 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.715188 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.716021 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.716001 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8008214d-c86a-4ff0-9838-54c39f5d2a81-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.767629 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.767571 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64k57\" (UniqueName: \"kubernetes.io/projected/8008214d-c86a-4ff0-9838-54c39f5d2a81-kube-api-access-64k57\") pod \"thanos-querier-6646454b57-j46xn\" (UID: \"8008214d-c86a-4ff0-9838-54c39f5d2a81\") " pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:30.954054 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:30.954032 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:31.061679 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:31.059610 4399 generic.go:356] "Generic (PLEG): container finished" podID="f6e52f66-5e0e-4cbb-852d-fd4aa259d271" containerID="200be2b96e73209427182dcb72c205d70b5a618db5d2372f323a9ce07b3c497b" exitCode=0 Oct 31 14:46:31.061679 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:31.059733 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2jxj6" event={"ID":"f6e52f66-5e0e-4cbb-852d-fd4aa259d271","Type":"ContainerDied","Data":"200be2b96e73209427182dcb72c205d70b5a618db5d2372f323a9ce07b3c497b"} Oct 31 14:46:31.072730 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:31.071790 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" event={"ID":"166f6dae-eda8-4568-8183-20db7b0dd0b6","Type":"ContainerStarted","Data":"96167d4ef969c02680a4397296f655a2338cd3b734f395ea5f896795480531c3"} Oct 31 14:46:31.291620 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:31.291564 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-gz5c7" podStartSLOduration=1.984592445 podStartE2EDuration="4.291547062s" totalImagesPullingTime="2.306954617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:31.290022982 +0000 UTC m=+34.501264394" watchObservedRunningTime="2026-10-31 14:46:31.291547062 +0000 UTC m=+34.502788473" Oct 31 14:46:31.459970 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:31.459934 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6646454b57-j46xn"] Oct 31 14:46:31.463224 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:31.463199 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8008214d_c86a_4ff0_9838_54c39f5d2a81.slice/crio-8d2137410ba857beb0afa1edc53e4c88a5029201fd36602ba9ecd1f147de9eb3 WatchSource:0}: Error finding container 8d2137410ba857beb0afa1edc53e4c88a5029201fd36602ba9ecd1f147de9eb3: Status 404 returned error can't find the container with id 8d2137410ba857beb0afa1edc53e4c88a5029201fd36602ba9ecd1f147de9eb3 Oct 31 14:46:32.074057 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.074026 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" event={"ID":"8008214d-c86a-4ff0-9838-54c39f5d2a81","Type":"ContainerStarted","Data":"8d2137410ba857beb0afa1edc53e4c88a5029201fd36602ba9ecd1f147de9eb3"} Oct 31 14:46:32.076727 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.076678 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2jxj6" event={"ID":"f6e52f66-5e0e-4cbb-852d-fd4aa259d271","Type":"ContainerStarted","Data":"16124bebba513ccb47472ddb2b3809e428e8a2c6fae276be225a87a3bad9dea0"} Oct 31 14:46:32.076869 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.076755 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2jxj6" event={"ID":"f6e52f66-5e0e-4cbb-852d-fd4aa259d271","Type":"ContainerStarted","Data":"5c6e723bb3aadf69e162468cae2421a911169c4fb6b1368ea98974cf959d1cd0"} Oct 31 14:46:32.197198 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.197146 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2jxj6" podStartSLOduration=2.745245504 podStartE2EDuration="5.19713069s" totalImagesPullingTime="2.357157119s" totalInitContainerRuntime="94.728067ms" isStatefulPod=true podCreationTimestamp="2026-10-31 14:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:32.195757759 +0000 UTC m=+35.406999171" watchObservedRunningTime="2026-10-31 14:46:32.19713069 +0000 UTC m=+35.408372104" Oct 31 14:46:32.480950 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.480917 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gxcct"] Oct 31 14:46:32.498727 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.498566 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:32.539140 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.539115 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gxcct"] Oct 31 14:46:32.551336 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.551299 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 14:46:32.551463 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.551425 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vjkjf\"" Oct 31 14:46:32.629999 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.629959 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a90dc3f-448b-4dd1-a571-6a9b1dc17349-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gxcct\" (UID: \"8a90dc3f-448b-4dd1-a571-6a9b1dc17349\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:32.630204 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.630134 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnr59\" (UniqueName: \"kubernetes.io/projected/8a90dc3f-448b-4dd1-a571-6a9b1dc17349-kube-api-access-qnr59\") pod \"monitoring-plugin-77f4f76594-gxcct\" (UID: \"8a90dc3f-448b-4dd1-a571-6a9b1dc17349\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:32.731669 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.731500 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a90dc3f-448b-4dd1-a571-6a9b1dc17349-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gxcct\" (UID: \"8a90dc3f-448b-4dd1-a571-6a9b1dc17349\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:32.731669 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.731647 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr59\" (UniqueName: \"kubernetes.io/projected/8a90dc3f-448b-4dd1-a571-6a9b1dc17349-kube-api-access-qnr59\") pod \"monitoring-plugin-77f4f76594-gxcct\" (UID: \"8a90dc3f-448b-4dd1-a571-6a9b1dc17349\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:32.742725 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.735623 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/8a90dc3f-448b-4dd1-a571-6a9b1dc17349-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-gxcct\" (UID: \"8a90dc3f-448b-4dd1-a571-6a9b1dc17349\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:32.789761 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.789735 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnr59\" (UniqueName: \"kubernetes.io/projected/8a90dc3f-448b-4dd1-a571-6a9b1dc17349-kube-api-access-qnr59\") pod \"monitoring-plugin-77f4f76594-gxcct\" (UID: \"8a90dc3f-448b-4dd1-a571-6a9b1dc17349\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:32.831613 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.831587 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:32.978853 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:32.978133 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-597c85d775-t24gx" Oct 31 14:46:33.233035 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.232998 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-gxcct"] Oct 31 14:46:33.237400 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:33.237363 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a90dc3f_448b_4dd1_a571_6a9b1dc17349.slice/crio-b1690a84dce681762870fa66ff105b696e36e559a0380e5f0ceb19c9b1807410 WatchSource:0}: Error finding container b1690a84dce681762870fa66ff105b696e36e559a0380e5f0ceb19c9b1807410: Status 404 returned error can't find the container with id b1690a84dce681762870fa66ff105b696e36e559a0380e5f0ceb19c9b1807410 Oct 31 14:46:33.414844 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.414810 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7585f46c4d-l65j7"] Oct 31 14:46:33.433185 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.433151 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.481481 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.481409 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7585f46c4d-l65j7"] Oct 31 14:46:33.485886 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.485828 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 14:46:33.486136 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.486112 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 14:46:33.529654 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.529619 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 14:46:33.529998 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.529975 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 14:46:33.535139 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.535093 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 14:46:33.535686 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.535633 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g79w8\"" Oct 31 14:46:33.539147 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.539125 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.539237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.539172 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f6a15-9317-42d1-8783-95ea990102a2-serving-certs-ca-bundle\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.539237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.539195 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-secret-telemeter-client\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.539237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.539213 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a7f6a15-9317-42d1-8783-95ea990102a2-metrics-client-ca\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.539338 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.539244 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-federate-client-tls\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.539338 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.539277 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-telemeter-client-tls\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.539338 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.539307 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f6a15-9317-42d1-8783-95ea990102a2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.539338 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.539325 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqkgg\" (UniqueName: \"kubernetes.io/projected/3a7f6a15-9317-42d1-8783-95ea990102a2-kube-api-access-xqkgg\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.583819 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.579978 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 14:46:33.641564 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.640526 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-secret-telemeter-client\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.641564 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.640572 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a7f6a15-9317-42d1-8783-95ea990102a2-metrics-client-ca\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.641564 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.640606 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-federate-client-tls\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.641564 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.640646 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-telemeter-client-tls\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.641564 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.640686 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f6a15-9317-42d1-8783-95ea990102a2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.641564 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.640740 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkgg\" (UniqueName: \"kubernetes.io/projected/3a7f6a15-9317-42d1-8783-95ea990102a2-kube-api-access-xqkgg\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.641564 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.640771 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.641564 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.640814 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f6a15-9317-42d1-8783-95ea990102a2-serving-certs-ca-bundle\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.642517 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.642166 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f6a15-9317-42d1-8783-95ea990102a2-serving-certs-ca-bundle\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.642846 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.642819 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a7f6a15-9317-42d1-8783-95ea990102a2-metrics-client-ca\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.643662 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.643634 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a7f6a15-9317-42d1-8783-95ea990102a2-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.644959 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.644934 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-secret-telemeter-client\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.645772 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.645749 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-federate-client-tls\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.646593 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.646551 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-telemeter-client-tls\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.647868 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.647844 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a7f6a15-9317-42d1-8783-95ea990102a2-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.699590 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.699559 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkgg\" (UniqueName: \"kubernetes.io/projected/3a7f6a15-9317-42d1-8783-95ea990102a2-kube-api-access-xqkgg\") pod \"telemeter-client-7585f46c4d-l65j7\" (UID: \"3a7f6a15-9317-42d1-8783-95ea990102a2\") " pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:33.803753 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:33.803663 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" Oct 31 14:46:34.088840 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:34.088728 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" event={"ID":"8a90dc3f-448b-4dd1-a571-6a9b1dc17349","Type":"ContainerStarted","Data":"b1690a84dce681762870fa66ff105b696e36e559a0380e5f0ceb19c9b1807410"} Oct 31 14:46:34.320348 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:34.320310 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7585f46c4d-l65j7"] Oct 31 14:46:35.096552 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:35.096376 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" event={"ID":"8008214d-c86a-4ff0-9838-54c39f5d2a81","Type":"ContainerStarted","Data":"356cfeec3678fa4399283caec3b3cf3317f586a3b9a22ecb4162d34d55988e3e"} Oct 31 14:46:35.096552 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:35.096461 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" event={"ID":"8008214d-c86a-4ff0-9838-54c39f5d2a81","Type":"ContainerStarted","Data":"1e614ab528e4a4c01a595907e24e9389a531fbc3a0841ef1edf7fa9a439e4500"} Oct 31 14:46:35.096552 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:35.096498 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" event={"ID":"8008214d-c86a-4ff0-9838-54c39f5d2a81","Type":"ContainerStarted","Data":"bf685b5f1648c9f14de45d69d1981a5f052b9b0f20591050283e2b1db277ce16"} Oct 31 14:46:35.097941 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:35.097913 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" event={"ID":"3a7f6a15-9317-42d1-8783-95ea990102a2","Type":"ContainerStarted","Data":"0ead4f85a902c64d3aa447bedb33f54516d230e3538da6bfac5547d1f50843cc"} Oct 31 14:46:36.105290 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.105182 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" event={"ID":"8a90dc3f-448b-4dd1-a571-6a9b1dc17349","Type":"ContainerStarted","Data":"e354cb9ac387d8c4113fc0a36d359ca2727d4f467e5633f105fc0c06bd08ea42"} Oct 31 14:46:36.105832 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.105421 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:36.111492 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.111469 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" Oct 31 14:46:36.336158 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.336088 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-gxcct" podStartSLOduration=1.823548639 podStartE2EDuration="4.336069253s" totalImagesPullingTime="2.512520614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:36.223946303 +0000 UTC m=+39.435187716" watchObservedRunningTime="2026-10-31 14:46:36.336069253 +0000 UTC m=+39.547310667" Oct 31 14:46:36.851544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.850287 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 14:46:36.877990 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.877958 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.933316 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.933287 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 14:46:36.933645 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.933453 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fspng\"" Oct 31 14:46:36.933645 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.933557 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 14:46:36.933645 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.933616 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 14:46:36.975520 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.975265 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1aauuj6ttrpu2\"" Oct 31 14:46:36.975520 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.975366 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 14:46:36.975520 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.975478 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 14:46:36.975520 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.975487 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 14:46:36.976736 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976604 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.976736 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976644 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config-out\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.976736 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976669 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.976736 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976723 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976749 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976775 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976800 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976828 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976853 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976888 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976922 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976960 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l57jv\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-kube-api-access-l57jv\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.976995 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.977028 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.977063 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.977106 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.977130 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-web-config\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.977167 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.977195 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.977447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.977217 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:36.979659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:36.979607 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 14:46:37.077996 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.077972 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078133 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078004 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config-out\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078133 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078030 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078133 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078072 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078133 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078101 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078133 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078132 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078398 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078157 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078398 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078184 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078398 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078205 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078398 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078227 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078398 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078259 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078398 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078294 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57jv\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-kube-api-access-l57jv\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078398 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078328 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078398 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078361 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078400 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078446 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078471 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-web-config\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078517 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078550 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078573 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.078830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.078811 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.080445 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.079304 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.080445 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.080233 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.080445 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.080300 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.080445 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.080389 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.080743 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.080646 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.081268 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.081112 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 14:46:37.081268 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.081165 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 14:46:37.083553 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.083452 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 14:46:37.085994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.084378 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 14:46:37.085994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.085762 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.086580 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.086267 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 14:46:37.086580 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.086333 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 14:46:37.086580 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.086473 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 14:46:37.086580 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.086579 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 14:46:37.086858 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.086641 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 14:46:37.089406 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.089388 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.094864 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.091506 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.094864 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.092503 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.094864 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.093710 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.094864 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.094758 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config-out\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.095775 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.095726 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.097636 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.097590 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.099319 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.099299 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.099920 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.099869 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-web-config\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.100137 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.100114 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.102221 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.102182 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.102897 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.102875 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.116943 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.116914 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" event={"ID":"8008214d-c86a-4ff0-9838-54c39f5d2a81","Type":"ContainerStarted","Data":"3784e5bee86fd04d35adbe64843f706c3258f8a69aa7ea4867ee939bbd7f30e9"} Oct 31 14:46:37.117253 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.116989 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" event={"ID":"8008214d-c86a-4ff0-9838-54c39f5d2a81","Type":"ContainerStarted","Data":"2b393a0ee1466723cccc482d7e171c39efbd1ea245c66eade604032a952e52f5"} Oct 31 14:46:37.117253 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.117029 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" event={"ID":"8008214d-c86a-4ff0-9838-54c39f5d2a81","Type":"ContainerStarted","Data":"d47b0d721f033bd84e6b14bf034c6ee3269302a45c68cc076e57bca3d1d7294e"} Oct 31 14:46:37.117253 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.117070 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:37.135885 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.135862 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57jv\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-kube-api-access-l57jv\") pod \"prometheus-k8s-0\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.265545 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.265413 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" podStartSLOduration=2.366393265 podStartE2EDuration="7.265395563s" totalImagesPullingTime="4.899002298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:37.26379463 +0000 UTC m=+40.475036041" watchObservedRunningTime="2026-10-31 14:46:37.265395563 +0000 UTC m=+40.476636979" Oct 31 14:46:37.342029 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.342003 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:37.860664 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:37.860638 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 14:46:37.863832 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:46:37.863806 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364df9f7_c7c3_45b8_8ebe_d6080e46c97b.slice/crio-26e8a04dea232659565191b27fc9082e18bd48bd288702e6cca0fce1ec75ab84 WatchSource:0}: Error finding container 26e8a04dea232659565191b27fc9082e18bd48bd288702e6cca0fce1ec75ab84: Status 404 returned error can't find the container with id 26e8a04dea232659565191b27fc9082e18bd48bd288702e6cca0fce1ec75ab84 Oct 31 14:46:38.120481 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:38.120448 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerStarted","Data":"26e8a04dea232659565191b27fc9082e18bd48bd288702e6cca0fce1ec75ab84"} Oct 31 14:46:38.122024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:38.122002 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" event={"ID":"3a7f6a15-9317-42d1-8783-95ea990102a2","Type":"ContainerStarted","Data":"1ce275738ecd728217bf3047b57777a26d751198d30b2c954269e4ce9e72e38d"} Oct 31 14:46:40.131120 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:40.131086 4399 generic.go:356] "Generic (PLEG): container finished" podID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerID="823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1" exitCode=0 Oct 31 14:46:40.131581 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:40.131184 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerDied","Data":"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1"} Oct 31 14:46:40.133555 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:40.133533 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" event={"ID":"3a7f6a15-9317-42d1-8783-95ea990102a2","Type":"ContainerStarted","Data":"85bc557a127b9976fa860e5356dac0f75b16221db26e67ccac5aade680473939"} Oct 31 14:46:40.133639 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:40.133595 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" event={"ID":"3a7f6a15-9317-42d1-8783-95ea990102a2","Type":"ContainerStarted","Data":"4e1bb20405883b76647ee7c9e5840065b8a1998cf8c22506b3e7bc1193214c98"} Oct 31 14:46:40.391499 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:40.391403 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7585f46c4d-l65j7" podStartSLOduration=2.031226332 podStartE2EDuration="7.391388663s" totalImagesPullingTime="5.360162331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:40.390514074 +0000 UTC m=+43.601755486" watchObservedRunningTime="2026-10-31 14:46:40.391388663 +0000 UTC m=+43.602630075" Oct 31 14:46:43.129103 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:43.129075 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6646454b57-j46xn" Oct 31 14:46:44.150944 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:44.150918 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerStarted","Data":"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125"} Oct 31 14:46:45.157508 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:45.157475 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerStarted","Data":"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07"} Oct 31 14:46:45.157897 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:45.157542 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerStarted","Data":"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2"} Oct 31 14:46:45.157897 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:45.157568 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerStarted","Data":"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65"} Oct 31 14:46:45.157897 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:45.157598 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerStarted","Data":"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89"} Oct 31 14:46:45.157897 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:45.157623 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerStarted","Data":"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c"} Oct 31 14:46:45.286214 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:45.286163 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.402214264 podStartE2EDuration="9.286149639s" totalImagesPullingTime="5.850766684s" totalInitContainerRuntime="33.168691ms" isStatefulPod=false podCreationTimestamp="2026-10-31 14:46:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:46:45.284619812 +0000 UTC m=+48.495861225" watchObservedRunningTime="2026-10-31 14:46:45.286149639 +0000 UTC m=+48.497391075" Oct 31 14:46:47.342711 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:47.342665 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:56.205381 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:46:56.205345 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:46:59.340925 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:59.340875 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:46:59.359360 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:46:59.359338 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:00.222235 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:00.222210 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:26.209752 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:47:26.209718 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:47:27.834062 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:27.834028 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 14:47:27.834575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:27.834371 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="prometheus" containerID="cri-o://0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" gracePeriod=600 Oct 31 14:47:27.834575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:27.834428 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy" containerID="cri-o://fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" gracePeriod=600 Oct 31 14:47:27.834575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:27.834492 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" gracePeriod=600 Oct 31 14:47:27.834575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:27.834464 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="config-reloader" containerID="cri-o://1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" gracePeriod=600 Oct 31 14:47:27.834575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:27.834529 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy-web" containerID="cri-o://24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" gracePeriod=600 Oct 31 14:47:27.834854 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:27.834487 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="thanos-sidecar" containerID="cri-o://aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" gracePeriod=600 Oct 31 14:47:28.078816 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.078791 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.216144 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216116 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-thanos-prometheus-http-client-file\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.216315 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216163 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-tls-assets\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-tls-assets\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.216315 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216206 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.216315 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216240 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-2\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.216488 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216385 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-trusted-ca-bundle\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.216488 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216427 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config-out\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.216595 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216498 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-metrics-client-ca\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.216595 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216548 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-metrics-client-certs\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.216595 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216575 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-kubelet-serving-ca-bundle\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216629 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-0\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216663 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-1\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216737 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-grpc-tls\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216791 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-web-config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-web-config\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216838 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-trusted-ca-bundle" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216856 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-kube-api-access-l57jv\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-kube-api-access-l57jv\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216870 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-2" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216888 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-tls\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216949 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.216975 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-db\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.217030 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-serving-certs-ca-bundle\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.217058 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.217116 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-kube-rbac-proxy\") pod \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\" (UID: \"364df9f7-c7c3-45b8-8ebe-d6080e46c97b\") " Oct 31 14:47:28.217447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.217196 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-1" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:47:28.218408 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.217608 4399 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.218408 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.217647 4399 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.218408 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.217664 4399 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.218408 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.217616 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-kubelet-serving-ca-bundle" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:47:28.219445 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.219025 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-0" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:47:28.220215 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.220136 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-metrics-client-ca" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:47:28.221369 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.221336 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-serving-certs-ca-bundle" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:47:28.222226 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.222087 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-kube-api-access-l57jv" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "kube-api-access-l57jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:47:28.222742 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.222717 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-grpc-tls" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:47:28.222992 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.222964 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-metrics-client-certs" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:47:28.223067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.223043 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-kube-rbac-proxy-web" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:47:28.223947 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.223930 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-thanos-prometheus-http-client-file" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:47:28.223947 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.223932 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-tls-assets" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:47:28.224522 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.224499 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:47:28.225833 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.225804 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config-out" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:47:28.225928 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.225856 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-tls" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:47:28.226329 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.226313 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-kube-rbac-proxy" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:47:28.227177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.227151 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-db" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:47:28.227810 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.227788 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-thanos-sidecar-tls" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:47:28.233017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.232993 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-web-config" pod "364df9f7-c7c3-45b8-8ebe-d6080e46c97b" (UID: "364df9f7-c7c3-45b8-8ebe-d6080e46c97b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:47:28.311810 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311777 4399 generic.go:356] "Generic (PLEG): container finished" podID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerID="ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" exitCode=0 Oct 31 14:47:28.311810 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311800 4399 generic.go:356] "Generic (PLEG): container finished" podID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerID="fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" exitCode=0 Oct 31 14:47:28.311810 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311808 4399 generic.go:356] "Generic (PLEG): container finished" podID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerID="24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" exitCode=0 Oct 31 14:47:28.311810 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311815 4399 generic.go:356] "Generic (PLEG): container finished" podID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerID="aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" exitCode=0 Oct 31 14:47:28.311810 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311820 4399 generic.go:356] "Generic (PLEG): container finished" podID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerID="1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" exitCode=0 Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311825 4399 generic.go:356] "Generic (PLEG): container finished" podID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerID="0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" exitCode=0 Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311828 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerDied","Data":"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07"} Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311851 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311921 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerDied","Data":"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2"} Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311964 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerDied","Data":"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65"} Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.311981 4399 scope.go:122] "RemoveContainer" containerID="ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.312006 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerDied","Data":"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89"} Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.312046 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerDied","Data":"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c"} Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.312086 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerDied","Data":"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125"} Oct 31 14:47:28.312167 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.312128 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"364df9f7-c7c3-45b8-8ebe-d6080e46c97b","Type":"ContainerDied","Data":"26e8a04dea232659565191b27fc9082e18bd48bd288702e6cca0fce1ec75ab84"} Oct 31 14:47:28.318121 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318096 4399 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config-out\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318130 4399 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-metrics-client-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318145 4399 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-metrics-client-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318160 4399 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318169 4399 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318177 4399 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-grpc-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318187 4399 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-web-config\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318195 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l57jv\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-kube-api-access-l57jv\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318205 4399 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318213 4399 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-config\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318226 4399 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-prometheus-k8s-db\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318239 4399 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318252 4399 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318260 4399 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-kube-rbac-proxy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318269 4399 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318282 4399 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-tls-assets\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.318544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.318296 4399 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/364df9f7-c7c3-45b8-8ebe-d6080e46c97b-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:47:28.319790 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.319775 4399 scope.go:122] "RemoveContainer" containerID="fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" Oct 31 14:47:28.326491 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.326477 4399 scope.go:122] "RemoveContainer" containerID="24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" Oct 31 14:47:28.333008 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.332988 4399 scope.go:122] "RemoveContainer" containerID="aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" Oct 31 14:47:28.339142 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.339129 4399 scope.go:122] "RemoveContainer" containerID="1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" Oct 31 14:47:28.345394 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.345377 4399 scope.go:122] "RemoveContainer" containerID="0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" Oct 31 14:47:28.352056 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.352040 4399 scope.go:122] "RemoveContainer" containerID="823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1" Oct 31 14:47:28.358351 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.358335 4399 scope.go:122] "RemoveContainer" containerID="ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" Oct 31 14:47:28.358628 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:47:28.358606 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": container with ID starting with ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07 not found: ID does not exist" containerID="ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" Oct 31 14:47:28.358674 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.358639 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07"} err="failed to get container status \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": rpc error: code = NotFound desc = could not find container \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": container with ID starting with ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07 not found: ID does not exist" Oct 31 14:47:28.358674 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.358668 4399 scope.go:122] "RemoveContainer" containerID="fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" Oct 31 14:47:28.358962 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:47:28.358945 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": container with ID starting with fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2 not found: ID does not exist" containerID="fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" Oct 31 14:47:28.359017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.358970 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2"} err="failed to get container status \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": rpc error: code = NotFound desc = could not find container \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": container with ID starting with fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2 not found: ID does not exist" Oct 31 14:47:28.359017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.358987 4399 scope.go:122] "RemoveContainer" containerID="24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" Oct 31 14:47:28.359238 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:47:28.359221 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": container with ID starting with 24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65 not found: ID does not exist" containerID="24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" Oct 31 14:47:28.359320 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.359241 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65"} err="failed to get container status \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": rpc error: code = NotFound desc = could not find container \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": container with ID starting with 24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65 not found: ID does not exist" Oct 31 14:47:28.359320 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.359256 4399 scope.go:122] "RemoveContainer" containerID="aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" Oct 31 14:47:28.359456 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:47:28.359442 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": container with ID starting with aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89 not found: ID does not exist" containerID="aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" Oct 31 14:47:28.359497 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.359458 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89"} err="failed to get container status \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": rpc error: code = NotFound desc = could not find container \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": container with ID starting with aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89 not found: ID does not exist" Oct 31 14:47:28.359497 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.359469 4399 scope.go:122] "RemoveContainer" containerID="1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" Oct 31 14:47:28.359724 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:47:28.359691 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": container with ID starting with 1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c not found: ID does not exist" containerID="1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" Oct 31 14:47:28.359790 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.359726 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c"} err="failed to get container status \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": rpc error: code = NotFound desc = could not find container \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": container with ID starting with 1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c not found: ID does not exist" Oct 31 14:47:28.359790 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.359738 4399 scope.go:122] "RemoveContainer" containerID="0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" Oct 31 14:47:28.359942 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:47:28.359926 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": container with ID starting with 0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125 not found: ID does not exist" containerID="0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" Oct 31 14:47:28.359980 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.359946 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125"} err="failed to get container status \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": rpc error: code = NotFound desc = could not find container \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": container with ID starting with 0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125 not found: ID does not exist" Oct 31 14:47:28.359980 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.359957 4399 scope.go:122] "RemoveContainer" containerID="823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1" Oct 31 14:47:28.360181 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:47:28.360162 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": container with ID starting with 823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1 not found: ID does not exist" containerID="823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1" Oct 31 14:47:28.360227 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.360183 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1"} err="failed to get container status \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": rpc error: code = NotFound desc = could not find container \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": container with ID starting with 823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1 not found: ID does not exist" Oct 31 14:47:28.360227 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.360194 4399 scope.go:122] "RemoveContainer" containerID="ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" Oct 31 14:47:28.360363 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.360348 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07"} err="failed to get container status \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": rpc error: code = NotFound desc = could not find container \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": container with ID starting with ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07 not found: ID does not exist" Oct 31 14:47:28.360401 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.360364 4399 scope.go:122] "RemoveContainer" containerID="fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" Oct 31 14:47:28.360575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.360558 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2"} err="failed to get container status \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": rpc error: code = NotFound desc = could not find container \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": container with ID starting with fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2 not found: ID does not exist" Oct 31 14:47:28.360575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.360574 4399 scope.go:122] "RemoveContainer" containerID="24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" Oct 31 14:47:28.360800 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.360785 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65"} err="failed to get container status \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": rpc error: code = NotFound desc = could not find container \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": container with ID starting with 24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65 not found: ID does not exist" Oct 31 14:47:28.360800 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.360800 4399 scope.go:122] "RemoveContainer" containerID="aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" Oct 31 14:47:28.361112 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361096 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89"} err="failed to get container status \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": rpc error: code = NotFound desc = could not find container \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": container with ID starting with aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89 not found: ID does not exist" Oct 31 14:47:28.361112 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361111 4399 scope.go:122] "RemoveContainer" containerID="1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" Oct 31 14:47:28.361307 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361291 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c"} err="failed to get container status \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": rpc error: code = NotFound desc = could not find container \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": container with ID starting with 1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c not found: ID does not exist" Oct 31 14:47:28.361348 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361307 4399 scope.go:122] "RemoveContainer" containerID="0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" Oct 31 14:47:28.361482 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361467 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125"} err="failed to get container status \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": rpc error: code = NotFound desc = could not find container \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": container with ID starting with 0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125 not found: ID does not exist" Oct 31 14:47:28.361522 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361482 4399 scope.go:122] "RemoveContainer" containerID="823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1" Oct 31 14:47:28.361675 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361660 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1"} err="failed to get container status \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": rpc error: code = NotFound desc = could not find container \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": container with ID starting with 823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1 not found: ID does not exist" Oct 31 14:47:28.361769 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361674 4399 scope.go:122] "RemoveContainer" containerID="ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" Oct 31 14:47:28.361873 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361857 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07"} err="failed to get container status \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": rpc error: code = NotFound desc = could not find container \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": container with ID starting with ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07 not found: ID does not exist" Oct 31 14:47:28.361923 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.361873 4399 scope.go:122] "RemoveContainer" containerID="fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" Oct 31 14:47:28.362075 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362056 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2"} err="failed to get container status \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": rpc error: code = NotFound desc = could not find container \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": container with ID starting with fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2 not found: ID does not exist" Oct 31 14:47:28.362113 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362077 4399 scope.go:122] "RemoveContainer" containerID="24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" Oct 31 14:47:28.362297 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362283 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65"} err="failed to get container status \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": rpc error: code = NotFound desc = could not find container \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": container with ID starting with 24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65 not found: ID does not exist" Oct 31 14:47:28.362336 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362298 4399 scope.go:122] "RemoveContainer" containerID="aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" Oct 31 14:47:28.362488 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362471 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89"} err="failed to get container status \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": rpc error: code = NotFound desc = could not find container \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": container with ID starting with aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89 not found: ID does not exist" Oct 31 14:47:28.362536 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362490 4399 scope.go:122] "RemoveContainer" containerID="1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" Oct 31 14:47:28.362683 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362668 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c"} err="failed to get container status \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": rpc error: code = NotFound desc = could not find container \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": container with ID starting with 1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c not found: ID does not exist" Oct 31 14:47:28.362736 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362683 4399 scope.go:122] "RemoveContainer" containerID="0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" Oct 31 14:47:28.362902 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362887 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125"} err="failed to get container status \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": rpc error: code = NotFound desc = could not find container \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": container with ID starting with 0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125 not found: ID does not exist" Oct 31 14:47:28.362949 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.362903 4399 scope.go:122] "RemoveContainer" containerID="823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1" Oct 31 14:47:28.363083 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363067 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1"} err="failed to get container status \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": rpc error: code = NotFound desc = could not find container \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": container with ID starting with 823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1 not found: ID does not exist" Oct 31 14:47:28.363130 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363083 4399 scope.go:122] "RemoveContainer" containerID="ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" Oct 31 14:47:28.363261 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363242 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07"} err="failed to get container status \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": rpc error: code = NotFound desc = could not find container \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": container with ID starting with ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07 not found: ID does not exist" Oct 31 14:47:28.363300 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363262 4399 scope.go:122] "RemoveContainer" containerID="fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" Oct 31 14:47:28.363476 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363460 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2"} err="failed to get container status \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": rpc error: code = NotFound desc = could not find container \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": container with ID starting with fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2 not found: ID does not exist" Oct 31 14:47:28.363518 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363477 4399 scope.go:122] "RemoveContainer" containerID="24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" Oct 31 14:47:28.363708 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363677 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65"} err="failed to get container status \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": rpc error: code = NotFound desc = could not find container \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": container with ID starting with 24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65 not found: ID does not exist" Oct 31 14:47:28.363753 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363716 4399 scope.go:122] "RemoveContainer" containerID="aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" Oct 31 14:47:28.363980 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363955 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89"} err="failed to get container status \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": rpc error: code = NotFound desc = could not find container \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": container with ID starting with aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89 not found: ID does not exist" Oct 31 14:47:28.364060 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.363981 4399 scope.go:122] "RemoveContainer" containerID="1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" Oct 31 14:47:28.364204 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.364184 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c"} err="failed to get container status \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": rpc error: code = NotFound desc = could not find container \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": container with ID starting with 1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c not found: ID does not exist" Oct 31 14:47:28.364279 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.364206 4399 scope.go:122] "RemoveContainer" containerID="0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" Oct 31 14:47:28.364403 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.364383 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125"} err="failed to get container status \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": rpc error: code = NotFound desc = could not find container \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": container with ID starting with 0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125 not found: ID does not exist" Oct 31 14:47:28.364445 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.364405 4399 scope.go:122] "RemoveContainer" containerID="823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1" Oct 31 14:47:28.364636 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.364614 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1"} err="failed to get container status \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": rpc error: code = NotFound desc = could not find container \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": container with ID starting with 823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1 not found: ID does not exist" Oct 31 14:47:28.364679 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.364639 4399 scope.go:122] "RemoveContainer" containerID="ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" Oct 31 14:47:28.364912 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.364896 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07"} err="failed to get container status \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": rpc error: code = NotFound desc = could not find container \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": container with ID starting with ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07 not found: ID does not exist" Oct 31 14:47:28.364990 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.364912 4399 scope.go:122] "RemoveContainer" containerID="fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" Oct 31 14:47:28.365123 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365108 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2"} err="failed to get container status \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": rpc error: code = NotFound desc = could not find container \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": container with ID starting with fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2 not found: ID does not exist" Oct 31 14:47:28.365159 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365123 4399 scope.go:122] "RemoveContainer" containerID="24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" Oct 31 14:47:28.365340 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365324 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65"} err="failed to get container status \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": rpc error: code = NotFound desc = could not find container \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": container with ID starting with 24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65 not found: ID does not exist" Oct 31 14:47:28.365389 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365340 4399 scope.go:122] "RemoveContainer" containerID="aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" Oct 31 14:47:28.365502 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365488 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89"} err="failed to get container status \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": rpc error: code = NotFound desc = could not find container \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": container with ID starting with aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89 not found: ID does not exist" Oct 31 14:47:28.365550 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365502 4399 scope.go:122] "RemoveContainer" containerID="1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" Oct 31 14:47:28.365734 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365716 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c"} err="failed to get container status \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": rpc error: code = NotFound desc = could not find container \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": container with ID starting with 1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c not found: ID does not exist" Oct 31 14:47:28.365771 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365735 4399 scope.go:122] "RemoveContainer" containerID="0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" Oct 31 14:47:28.365931 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365911 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125"} err="failed to get container status \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": rpc error: code = NotFound desc = could not find container \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": container with ID starting with 0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125 not found: ID does not exist" Oct 31 14:47:28.365931 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.365929 4399 scope.go:122] "RemoveContainer" containerID="823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1" Oct 31 14:47:28.366101 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366087 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1"} err="failed to get container status \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": rpc error: code = NotFound desc = could not find container \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": container with ID starting with 823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1 not found: ID does not exist" Oct 31 14:47:28.366101 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366100 4399 scope.go:122] "RemoveContainer" containerID="ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07" Oct 31 14:47:28.366282 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366262 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07"} err="failed to get container status \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": rpc error: code = NotFound desc = could not find container \"ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07\": container with ID starting with ecf7b3f55ba7f50901ef5526b00d4e8431b5297c7224ade0a774e987e5aaba07 not found: ID does not exist" Oct 31 14:47:28.366320 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366283 4399 scope.go:122] "RemoveContainer" containerID="fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2" Oct 31 14:47:28.366503 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366484 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2"} err="failed to get container status \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": rpc error: code = NotFound desc = could not find container \"fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2\": container with ID starting with fc4cad4db970dca2a1a67cedb50765afe18aa46b2e5c65d39b54a4c522df42c2 not found: ID does not exist" Oct 31 14:47:28.366546 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366503 4399 scope.go:122] "RemoveContainer" containerID="24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65" Oct 31 14:47:28.366736 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366716 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65"} err="failed to get container status \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": rpc error: code = NotFound desc = could not find container \"24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65\": container with ID starting with 24d3b1857fefb6fa5e5a48c45a5a2d4a4f759d6600fbe780c0ac5bd6cae54f65 not found: ID does not exist" Oct 31 14:47:28.366812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366737 4399 scope.go:122] "RemoveContainer" containerID="aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89" Oct 31 14:47:28.366961 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366946 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89"} err="failed to get container status \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": rpc error: code = NotFound desc = could not find container \"aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89\": container with ID starting with aff317a6eb0e809f837d7ae52052baac621e69195d11677fa6534c11e4950d89 not found: ID does not exist" Oct 31 14:47:28.367028 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.366962 4399 scope.go:122] "RemoveContainer" containerID="1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c" Oct 31 14:47:28.367174 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.367158 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c"} err="failed to get container status \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": rpc error: code = NotFound desc = could not find container \"1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c\": container with ID starting with 1f4f9e9aaa2de0691ab6bf8f0d026deaa2d1fc8708369923dcda4799e09d757c not found: ID does not exist" Oct 31 14:47:28.367237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.367175 4399 scope.go:122] "RemoveContainer" containerID="0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125" Oct 31 14:47:28.367394 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.367376 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125"} err="failed to get container status \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": rpc error: code = NotFound desc = could not find container \"0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125\": container with ID starting with 0a8fa1bf1b8e6667555b6bf4b7d7a2f6d566aa79cf6b69af4d05cb0224dc5125 not found: ID does not exist" Oct 31 14:47:28.367439 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.367394 4399 scope.go:122] "RemoveContainer" containerID="823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1" Oct 31 14:47:28.367646 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.367628 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1"} err="failed to get container status \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": rpc error: code = NotFound desc = could not find container \"823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1\": container with ID starting with 823a7873e94d86d62474880e4f763f01e9954884ebbaf74965d871a373ec6ba1 not found: ID does not exist" Oct 31 14:47:28.477264 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.477201 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 14:47:28.483017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.482998 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.510762 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511308 4399 cpu_manager.go:414] "containerMap: removing container" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="prometheus" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511323 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="prometheus" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511338 4399 cpu_manager.go:414] "containerMap: removing container" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="thanos-sidecar" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511345 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="thanos-sidecar" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511365 4399 cpu_manager.go:414] "containerMap: removing container" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="init-config-reloader" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511373 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="init-config-reloader" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511397 4399 cpu_manager.go:414] "containerMap: removing container" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy-web" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511404 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy-web" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511416 4399 cpu_manager.go:414] "containerMap: removing container" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511423 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511467 4399 cpu_manager.go:414] "containerMap: removing container" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy-thanos" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511475 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy-thanos" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511493 4399 cpu_manager.go:414] "containerMap: removing container" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="config-reloader" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511501 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="config-reloader" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511635 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy-thanos" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511654 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="config-reloader" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511664 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="prometheus" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511671 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="thanos-sidecar" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511686 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy-web" Oct 31 14:47:28.512952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.511712 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" containerName="kube-rbac-proxy" Oct 31 14:47:28.516317 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.516302 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.569477 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569448 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 14:47:28.569595 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569538 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 14:47:28.569663 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569619 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 14:47:28.569663 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569624 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 14:47:28.569663 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569630 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 14:47:28.569823 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569754 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 14:47:28.569823 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569758 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 14:47:28.569903 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569888 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 14:47:28.569946 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569931 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 14:47:28.570000 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.569984 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 14:47:28.570033 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.570009 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 14:47:28.570262 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.570244 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fspng\"" Oct 31 14:47:28.570262 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.570256 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1aauuj6ttrpu2\"" Oct 31 14:47:28.570262 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.570249 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 14:47:28.570476 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.570465 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 14:47:28.619302 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619272 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619418 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619314 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619477 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619399 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619477 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619448 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619572 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619512 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-config\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619572 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619540 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619572 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619558 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619734 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619577 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619734 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619687 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30e0121f-40e7-4a32-b0a7-e635216391a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619848 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619738 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 14:47:28.619848 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619785 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619848 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619826 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619980 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619875 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 14:47:28.619980 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619892 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30e0121f-40e7-4a32-b0a7-e635216391a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.619980 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619933 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.620070 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.619987 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.620102 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.620073 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.620165 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.620147 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.620222 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.620186 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sffn\" (UniqueName: \"kubernetes.io/projected/30e0121f-40e7-4a32-b0a7-e635216391a2-kube-api-access-4sffn\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.620258 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.620245 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.620293 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.620272 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.620331 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.620297 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.621234 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.621216 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 14:47:28.721001 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.720974 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721006 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sffn\" (UniqueName: \"kubernetes.io/projected/30e0121f-40e7-4a32-b0a7-e635216391a2-kube-api-access-4sffn\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721024 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721041 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721077 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721121 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721151 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721479 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721182 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721479 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721205 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721479 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721241 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-config\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721479 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721264 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721479 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721290 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721768 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721732 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721824 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721800 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.721933 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721911 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722066 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721992 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722066 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.721998 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722168 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722030 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722168 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722059 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30e0121f-40e7-4a32-b0a7-e635216391a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722270 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722169 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722270 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722208 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722270 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722265 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30e0121f-40e7-4a32-b0a7-e635216391a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722517 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722326 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722517 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722352 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722517 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722396 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722517 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722393 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.722757 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.722613 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.724449 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.724421 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.725132 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.725106 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-config\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.725750 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.725636 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/30e0121f-40e7-4a32-b0a7-e635216391a2-config-out\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.725927 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.725906 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/30e0121f-40e7-4a32-b0a7-e635216391a2-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.726108 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.726084 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.726241 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.726219 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.726303 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.726289 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-web-config\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.726622 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.726602 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/30e0121f-40e7-4a32-b0a7-e635216391a2-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.726723 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.726603 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.727159 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.727133 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.727659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.727614 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.727659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.727626 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/30e0121f-40e7-4a32-b0a7-e635216391a2-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.777949 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.777927 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sffn\" (UniqueName: \"kubernetes.io/projected/30e0121f-40e7-4a32-b0a7-e635216391a2-kube-api-access-4sffn\") pod \"prometheus-k8s-0\" (UID: \"30e0121f-40e7-4a32-b0a7-e635216391a2\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.832035 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.832011 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:28.896276 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:28.896242 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364df9f7-c7c3-45b8-8ebe-d6080e46c97b" path="/var/lib/kubelet/pods/364df9f7-c7c3-45b8-8ebe-d6080e46c97b/volumes" Oct 31 14:47:29.259673 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:29.259648 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 14:47:29.262918 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:47:29.262884 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e0121f_40e7_4a32_b0a7_e635216391a2.slice/crio-c07eb64fb23b8407b29fe21ee258750aa05b0a13aa5872f50e478d6c5686ea08 WatchSource:0}: Error finding container c07eb64fb23b8407b29fe21ee258750aa05b0a13aa5872f50e478d6c5686ea08: Status 404 returned error can't find the container with id c07eb64fb23b8407b29fe21ee258750aa05b0a13aa5872f50e478d6c5686ea08 Oct 31 14:47:29.318927 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:29.318902 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30e0121f-40e7-4a32-b0a7-e635216391a2","Type":"ContainerStarted","Data":"c07eb64fb23b8407b29fe21ee258750aa05b0a13aa5872f50e478d6c5686ea08"} Oct 31 14:47:30.323799 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:30.323760 4399 generic.go:356] "Generic (PLEG): container finished" podID="30e0121f-40e7-4a32-b0a7-e635216391a2" containerID="a2e3d1a8758ef8a0ac8dc5dc703a077d43fc8e0cb93c8f0e58732e5c8d151d5b" exitCode=0 Oct 31 14:47:30.323799 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:30.323804 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30e0121f-40e7-4a32-b0a7-e635216391a2","Type":"ContainerDied","Data":"a2e3d1a8758ef8a0ac8dc5dc703a077d43fc8e0cb93c8f0e58732e5c8d151d5b"} Oct 31 14:47:31.331194 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:31.331158 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30e0121f-40e7-4a32-b0a7-e635216391a2","Type":"ContainerStarted","Data":"4af727f7e5cc46d05ae827f6575da27827bd611753865e2ccabfcc7a35dfedf1"} Oct 31 14:47:31.331613 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:31.331219 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30e0121f-40e7-4a32-b0a7-e635216391a2","Type":"ContainerStarted","Data":"d5d0d44284f74ee8f51deae833c776d2bf8180922e37d723529833fccd718c77"} Oct 31 14:47:31.331613 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:31.331245 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30e0121f-40e7-4a32-b0a7-e635216391a2","Type":"ContainerStarted","Data":"ce2496c7ea1a844cd06f540be6837ecb818adb42bded6140bd1d1089eb13673f"} Oct 31 14:47:31.331613 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:31.331271 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30e0121f-40e7-4a32-b0a7-e635216391a2","Type":"ContainerStarted","Data":"9be89a03e41fd04a20037fb7e19d902d03d93bda4dd96896d40b05e7103a6fe1"} Oct 31 14:47:31.331613 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:31.331293 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30e0121f-40e7-4a32-b0a7-e635216391a2","Type":"ContainerStarted","Data":"28bb5a32bbd21c006357b504768710c2d18154d0341eacf4d2a9c24632bf9232"} Oct 31 14:47:31.331613 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:31.331325 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"30e0121f-40e7-4a32-b0a7-e635216391a2","Type":"ContainerStarted","Data":"124ec31bc353422767bc63d7b42212f49e3e5962c5c6dac30803a0d4c9a4b544"} Oct 31 14:47:31.459489 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:31.459431 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.440863107 podStartE2EDuration="3.459415584s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.552477ms" isStatefulPod=false podCreationTimestamp="2026-10-31 14:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:47:31.457885018 +0000 UTC m=+94.669126461" watchObservedRunningTime="2026-10-31 14:47:31.459415584 +0000 UTC m=+94.670656996" Oct 31 14:47:33.832951 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:47:33.832908 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:47:56.204848 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:47:56.204802 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:48:26.205721 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:48:26.205667 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:48:28.832880 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:48:28.832841 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:48:28.848764 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:48:28.848737 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:48:29.544554 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:48:29.544528 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 14:48:56.204687 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:48:56.204652 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:49:09.977795 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:09.977764 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj"] Oct 31 14:49:09.982096 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:09.982067 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.037179 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.037151 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 14:49:10.037347 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.037329 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 14:49:10.037411 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.037381 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 14:49:10.083039 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.083006 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-webhook-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.083165 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.083066 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6kr\" (UniqueName: \"kubernetes.io/projected/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-kube-api-access-9g6kr\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.083231 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.083204 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-tmp\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.083272 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.083252 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.083359 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.083343 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-manifests\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.083399 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.083385 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-metrics-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.084292 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.084263 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 14:49:10.084400 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.084294 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 14:49:10.084400 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.084317 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rxbg7\"" Oct 31 14:49:10.086389 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.086369 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj"] Oct 31 14:49:10.183931 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.183908 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6kr\" (UniqueName: \"kubernetes.io/projected/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-kube-api-access-9g6kr\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.184111 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.183946 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-tmp\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.184111 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.183963 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.184111 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.184091 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-manifests\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.184266 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.184125 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-metrics-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.184266 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.184152 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-webhook-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.184505 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.184468 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-manifests\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.186422 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.186397 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-tmp\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.186642 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.186623 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.186730 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.186626 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-webhook-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.187141 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.187119 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-metrics-cert\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.242988 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.242942 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6kr\" (UniqueName: \"kubernetes.io/projected/a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa-kube-api-access-9g6kr\") pod \"opendatahub-operator-controller-manager-c95c77854-gvwgj\" (UID: \"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.296061 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.296041 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:10.803742 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:10.803548 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj"] Oct 31 14:49:10.806420 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:49:10.806391 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61e0cc8_3aae_4f5b_b9ee_2bdc838f8caa.slice/crio-17f0f1fe222ac9413af43b37250c3014ec09c14ef11d82e043feed7a161d67b0 WatchSource:0}: Error finding container 17f0f1fe222ac9413af43b37250c3014ec09c14ef11d82e043feed7a161d67b0: Status 404 returned error can't find the container with id 17f0f1fe222ac9413af43b37250c3014ec09c14ef11d82e043feed7a161d67b0 Oct 31 14:49:11.682041 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:11.682005 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" event={"ID":"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa","Type":"ContainerStarted","Data":"17f0f1fe222ac9413af43b37250c3014ec09c14ef11d82e043feed7a161d67b0"} Oct 31 14:49:15.698213 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:15.698179 4399 generic.go:356] "Generic (PLEG): container finished" podID="a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa" containerID="4211b373d73a71789c97bb37663df1d0177978cca8beaf15fa1a46246b17f3a9" exitCode=0 Oct 31 14:49:15.698562 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:15.698223 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" event={"ID":"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa","Type":"ContainerDied","Data":"4211b373d73a71789c97bb37663df1d0177978cca8beaf15fa1a46246b17f3a9"} Oct 31 14:49:16.703956 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:16.703922 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" event={"ID":"a61e0cc8-3aae-4f5b-b9ee-2bdc838f8caa","Type":"ContainerStarted","Data":"b733d092edd8820ca0518365202d4420bf7535118aecd0a6c0d70b7dd8ac7a37"} Oct 31 14:49:16.704316 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:16.704059 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:16.934668 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:16.934551 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" podStartSLOduration=2.332017719 podStartE2EDuration="7.93453737s" totalImagesPullingTime="5.134440307s" totalInitContainerRuntime="468.079344ms" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:16.885097433 +0000 UTC m=+200.096338855" watchObservedRunningTime="2026-10-31 14:49:16.93453737 +0000 UTC m=+200.145778782" Oct 31 14:49:26.206116 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:49:26.206080 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:49:27.709507 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:27.709482 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c95c77854-gvwgj" Oct 31 14:49:29.494500 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.494460 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq"] Oct 31 14:49:29.502361 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.502346 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" Oct 31 14:49:29.534923 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.534899 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d3f0284-4cfd-465b-9ea9-a391c2c5d468-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4vnjq\" (UID: \"3d3f0284-4cfd-465b-9ea9-a391c2c5d468\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" Oct 31 14:49:29.535048 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.534935 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c2dx\" (UniqueName: \"kubernetes.io/projected/3d3f0284-4cfd-465b-9ea9-a391c2c5d468-kube-api-access-2c2dx\") pod \"cert-manager-cainjector-5c57bbd4fc-4vnjq\" (UID: \"3d3f0284-4cfd-465b-9ea9-a391c2c5d468\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" Oct 31 14:49:29.554304 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.554280 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq"] Oct 31 14:49:29.602562 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.602543 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 14:49:29.602689 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.602629 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-96tvm\"" Oct 31 14:49:29.602689 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.602641 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 14:49:29.635985 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.635965 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c2dx\" (UniqueName: \"kubernetes.io/projected/3d3f0284-4cfd-465b-9ea9-a391c2c5d468-kube-api-access-2c2dx\") pod \"cert-manager-cainjector-5c57bbd4fc-4vnjq\" (UID: \"3d3f0284-4cfd-465b-9ea9-a391c2c5d468\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" Oct 31 14:49:29.636084 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.636059 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d3f0284-4cfd-465b-9ea9-a391c2c5d468-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4vnjq\" (UID: \"3d3f0284-4cfd-465b-9ea9-a391c2c5d468\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" Oct 31 14:49:29.693585 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.693559 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3d3f0284-4cfd-465b-9ea9-a391c2c5d468-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-4vnjq\" (UID: \"3d3f0284-4cfd-465b-9ea9-a391c2c5d468\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" Oct 31 14:49:29.694245 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.694226 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c2dx\" (UniqueName: \"kubernetes.io/projected/3d3f0284-4cfd-465b-9ea9-a391c2c5d468-kube-api-access-2c2dx\") pod \"cert-manager-cainjector-5c57bbd4fc-4vnjq\" (UID: \"3d3f0284-4cfd-465b-9ea9-a391c2c5d468\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" Oct 31 14:49:29.828472 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:29.828411 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" Oct 31 14:49:30.208835 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:30.208805 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq"] Oct 31 14:49:30.212860 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:49:30.212834 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d3f0284_4cfd_465b_9ea9_a391c2c5d468.slice/crio-d23b01fb4ea7ff536c2af20100cf2e79667388542a66c65ddd28d1b9f5586b25 WatchSource:0}: Error finding container d23b01fb4ea7ff536c2af20100cf2e79667388542a66c65ddd28d1b9f5586b25: Status 404 returned error can't find the container with id d23b01fb4ea7ff536c2af20100cf2e79667388542a66c65ddd28d1b9f5586b25 Oct 31 14:49:30.755481 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:30.755448 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" event={"ID":"3d3f0284-4cfd-465b-9ea9-a391c2c5d468","Type":"ContainerStarted","Data":"d23b01fb4ea7ff536c2af20100cf2e79667388542a66c65ddd28d1b9f5586b25"} Oct 31 14:49:34.774513 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:34.774482 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" event={"ID":"3d3f0284-4cfd-465b-9ea9-a391c2c5d468","Type":"ContainerStarted","Data":"9189dd3d907774f51152ed9360627599f7c7f9923cf585ab9637e32e72d922a3"} Oct 31 14:49:34.903386 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:34.903334 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-4vnjq" podStartSLOduration=2.060625851 podStartE2EDuration="5.903319014s" totalImagesPullingTime="3.842693163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:34.901737438 +0000 UTC m=+218.112978850" watchObservedRunningTime="2026-10-31 14:49:34.903319014 +0000 UTC m=+218.114560425" Oct 31 14:49:36.881132 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:36.881101 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-5cxp2"] Oct 31 14:49:36.884548 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:36.884529 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-5cxp2" Oct 31 14:49:36.994403 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:36.994376 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-29hlw\"" Oct 31 14:49:36.995089 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:36.995062 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b86c3447-9ef4-4f37-8e31-5d5106d54aae-bound-sa-token\") pod \"cert-manager-597b95cb84-5cxp2\" (UID: \"b86c3447-9ef4-4f37-8e31-5d5106d54aae\") " pod="cert-manager/cert-manager-597b95cb84-5cxp2" Oct 31 14:49:36.995234 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:36.995152 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98twq\" (UniqueName: \"kubernetes.io/projected/b86c3447-9ef4-4f37-8e31-5d5106d54aae-kube-api-access-98twq\") pod \"cert-manager-597b95cb84-5cxp2\" (UID: \"b86c3447-9ef4-4f37-8e31-5d5106d54aae\") " pod="cert-manager/cert-manager-597b95cb84-5cxp2" Oct 31 14:49:36.995846 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:36.995827 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-5cxp2"] Oct 31 14:49:37.095986 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:37.095952 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b86c3447-9ef4-4f37-8e31-5d5106d54aae-bound-sa-token\") pod \"cert-manager-597b95cb84-5cxp2\" (UID: \"b86c3447-9ef4-4f37-8e31-5d5106d54aae\") " pod="cert-manager/cert-manager-597b95cb84-5cxp2" Oct 31 14:49:37.096126 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:37.096000 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98twq\" (UniqueName: \"kubernetes.io/projected/b86c3447-9ef4-4f37-8e31-5d5106d54aae-kube-api-access-98twq\") pod \"cert-manager-597b95cb84-5cxp2\" (UID: \"b86c3447-9ef4-4f37-8e31-5d5106d54aae\") " pod="cert-manager/cert-manager-597b95cb84-5cxp2" Oct 31 14:49:37.152513 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:37.152441 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98twq\" (UniqueName: \"kubernetes.io/projected/b86c3447-9ef4-4f37-8e31-5d5106d54aae-kube-api-access-98twq\") pod \"cert-manager-597b95cb84-5cxp2\" (UID: \"b86c3447-9ef4-4f37-8e31-5d5106d54aae\") " pod="cert-manager/cert-manager-597b95cb84-5cxp2" Oct 31 14:49:37.152513 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:37.152501 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b86c3447-9ef4-4f37-8e31-5d5106d54aae-bound-sa-token\") pod \"cert-manager-597b95cb84-5cxp2\" (UID: \"b86c3447-9ef4-4f37-8e31-5d5106d54aae\") " pod="cert-manager/cert-manager-597b95cb84-5cxp2" Oct 31 14:49:37.196433 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:37.196410 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-5cxp2" Oct 31 14:49:37.562843 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:37.562818 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-5cxp2"] Oct 31 14:49:37.566129 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:49:37.566101 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb86c3447_9ef4_4f37_8e31_5d5106d54aae.slice/crio-94a9df88ae1b905c5db6acd941cc1bb3cdce8eab6f54b977b8a79ae9269562a8 WatchSource:0}: Error finding container 94a9df88ae1b905c5db6acd941cc1bb3cdce8eab6f54b977b8a79ae9269562a8: Status 404 returned error can't find the container with id 94a9df88ae1b905c5db6acd941cc1bb3cdce8eab6f54b977b8a79ae9269562a8 Oct 31 14:49:37.787809 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:37.787773 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-5cxp2" event={"ID":"b86c3447-9ef4-4f37-8e31-5d5106d54aae","Type":"ContainerStarted","Data":"0af38c44ea7582021107317e0157dc4523c55c1ae28a0b193fdb873633a17523"} Oct 31 14:49:37.788001 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:37.787824 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-5cxp2" event={"ID":"b86c3447-9ef4-4f37-8e31-5d5106d54aae","Type":"ContainerStarted","Data":"94a9df88ae1b905c5db6acd941cc1bb3cdce8eab6f54b977b8a79ae9269562a8"} Oct 31 14:49:37.900135 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:37.900028 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-5cxp2" podStartSLOduration=1.900010145 podStartE2EDuration="1.900010145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:37.899605893 +0000 UTC m=+221.110847306" watchObservedRunningTime="2026-10-31 14:49:37.900010145 +0000 UTC m=+221.111251559" Oct 31 14:49:45.549621 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.549588 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs"] Oct 31 14:49:45.552975 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.552960 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" Oct 31 14:49:45.605805 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.605777 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 14:49:45.655240 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.655212 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 14:49:45.655240 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.655214 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6xfnp\"" Oct 31 14:49:45.656591 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.656574 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs"] Oct 31 14:49:45.668050 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.668026 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpcv\" (UniqueName: \"kubernetes.io/projected/338dae66-850f-4bbc-ae9e-da170cf58631-kube-api-access-qhpcv\") pod \"limitador-operator-controller-manager-5778bb4449-s7mjs\" (UID: \"338dae66-850f-4bbc-ae9e-da170cf58631\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" Oct 31 14:49:45.769429 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.769398 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpcv\" (UniqueName: \"kubernetes.io/projected/338dae66-850f-4bbc-ae9e-da170cf58631-kube-api-access-qhpcv\") pod \"limitador-operator-controller-manager-5778bb4449-s7mjs\" (UID: \"338dae66-850f-4bbc-ae9e-da170cf58631\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" Oct 31 14:49:45.829938 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.829868 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpcv\" (UniqueName: \"kubernetes.io/projected/338dae66-850f-4bbc-ae9e-da170cf58631-kube-api-access-qhpcv\") pod \"limitador-operator-controller-manager-5778bb4449-s7mjs\" (UID: \"338dae66-850f-4bbc-ae9e-da170cf58631\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" Oct 31 14:49:45.864947 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:45.864906 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" Oct 31 14:49:46.302811 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.302777 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs"] Oct 31 14:49:46.312507 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:49:46.312470 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338dae66_850f_4bbc_ae9e_da170cf58631.slice/crio-9c8aad7ce64fa92ef78ced5b6207d19443f1b1b77fcd67aef602bdd5523810ea WatchSource:0}: Error finding container 9c8aad7ce64fa92ef78ced5b6207d19443f1b1b77fcd67aef602bdd5523810ea: Status 404 returned error can't find the container with id 9c8aad7ce64fa92ef78ced5b6207d19443f1b1b77fcd67aef602bdd5523810ea Oct 31 14:49:46.368292 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.368255 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-9d7844885-hs7vq"] Oct 31 14:49:46.372738 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.372718 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.428193 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.427901 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 14:49:46.428653 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.428626 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9d7844885-hs7vq"] Oct 31 14:49:46.474474 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.474437 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e742a6-7f45-426c-8b68-c8819d5d294e-tls-certs\") pod \"kube-auth-proxy-9d7844885-hs7vq\" (UID: \"69e742a6-7f45-426c-8b68-c8819d5d294e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.474474 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.474473 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg4h5\" (UniqueName: \"kubernetes.io/projected/69e742a6-7f45-426c-8b68-c8819d5d294e-kube-api-access-cg4h5\") pod \"kube-auth-proxy-9d7844885-hs7vq\" (UID: \"69e742a6-7f45-426c-8b68-c8819d5d294e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.474674 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.474503 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e742a6-7f45-426c-8b68-c8819d5d294e-tmp\") pod \"kube-auth-proxy-9d7844885-hs7vq\" (UID: \"69e742a6-7f45-426c-8b68-c8819d5d294e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.475849 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.475805 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xsph8\"" Oct 31 14:49:46.475961 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.475857 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 14:49:46.475961 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.475870 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 14:49:46.475961 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.475859 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 14:49:46.575594 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.575506 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e742a6-7f45-426c-8b68-c8819d5d294e-tmp\") pod \"kube-auth-proxy-9d7844885-hs7vq\" (UID: \"69e742a6-7f45-426c-8b68-c8819d5d294e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.576006 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.575618 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e742a6-7f45-426c-8b68-c8819d5d294e-tls-certs\") pod \"kube-auth-proxy-9d7844885-hs7vq\" (UID: \"69e742a6-7f45-426c-8b68-c8819d5d294e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.576006 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.575649 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg4h5\" (UniqueName: \"kubernetes.io/projected/69e742a6-7f45-426c-8b68-c8819d5d294e-kube-api-access-cg4h5\") pod \"kube-auth-proxy-9d7844885-hs7vq\" (UID: \"69e742a6-7f45-426c-8b68-c8819d5d294e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.578025 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.577999 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69e742a6-7f45-426c-8b68-c8819d5d294e-tmp\") pod \"kube-auth-proxy-9d7844885-hs7vq\" (UID: \"69e742a6-7f45-426c-8b68-c8819d5d294e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.578143 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.578113 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69e742a6-7f45-426c-8b68-c8819d5d294e-tls-certs\") pod \"kube-auth-proxy-9d7844885-hs7vq\" (UID: \"69e742a6-7f45-426c-8b68-c8819d5d294e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.632122 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.632090 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg4h5\" (UniqueName: \"kubernetes.io/projected/69e742a6-7f45-426c-8b68-c8819d5d294e-kube-api-access-cg4h5\") pod \"kube-auth-proxy-9d7844885-hs7vq\" (UID: \"69e742a6-7f45-426c-8b68-c8819d5d294e\") " pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.685294 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.685270 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" Oct 31 14:49:46.830644 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:46.830553 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" event={"ID":"338dae66-850f-4bbc-ae9e-da170cf58631","Type":"ContainerStarted","Data":"9c8aad7ce64fa92ef78ced5b6207d19443f1b1b77fcd67aef602bdd5523810ea"} Oct 31 14:49:47.056839 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:47.056811 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-9d7844885-hs7vq"] Oct 31 14:49:47.060189 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:49:47.060163 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e742a6_7f45_426c_8b68_c8819d5d294e.slice/crio-a46046a0cd8881b09b7a0baa0b9b7fd33e6567a8c4228f24514524e70b46ecfc WatchSource:0}: Error finding container a46046a0cd8881b09b7a0baa0b9b7fd33e6567a8c4228f24514524e70b46ecfc: Status 404 returned error can't find the container with id a46046a0cd8881b09b7a0baa0b9b7fd33e6567a8c4228f24514524e70b46ecfc Oct 31 14:49:47.835190 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:47.835155 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" event={"ID":"69e742a6-7f45-426c-8b68-c8819d5d294e","Type":"ContainerStarted","Data":"a46046a0cd8881b09b7a0baa0b9b7fd33e6567a8c4228f24514524e70b46ecfc"} Oct 31 14:49:50.848870 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:50.848834 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" event={"ID":"338dae66-850f-4bbc-ae9e-da170cf58631","Type":"ContainerStarted","Data":"cf68c516a8ebe591b9883fe5ebf6698534b40faf69dc490f9a6626e5046cc0ac"} Oct 31 14:49:50.849311 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:50.848955 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" Oct 31 14:49:50.850577 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:50.850557 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" event={"ID":"69e742a6-7f45-426c-8b68-c8819d5d294e","Type":"ContainerStarted","Data":"f9519787f09fb50a60736f3f81bef1a85e5f9954d0febdba5ca03be47c28ee26"} Oct 31 14:49:51.076551 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:51.076499 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-9d7844885-hs7vq" podStartSLOduration=1.987905906 podStartE2EDuration="5.076482134s" totalImagesPullingTime="3.088576228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:51.076433606 +0000 UTC m=+234.287675029" watchObservedRunningTime="2026-10-31 14:49:51.076482134 +0000 UTC m=+234.287723549" Oct 31 14:49:51.077030 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:49:51.076947 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" podStartSLOduration=2.385734806 podStartE2EDuration="6.076936668s" totalImagesPullingTime="3.691201862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:49:50.962974777 +0000 UTC m=+234.174216189" watchObservedRunningTime="2026-10-31 14:49:51.076936668 +0000 UTC m=+234.288178080" Oct 31 14:49:56.206349 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:49:56.206312 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:50:01.856468 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:01.856434 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-s7mjs" Oct 31 14:50:02.082551 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.082521 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hdchk"] Oct 31 14:50:02.087136 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.087119 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:02.113256 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.113203 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2qt\" (UniqueName: \"kubernetes.io/projected/64920c2d-f5e0-4370-95b9-fd7861185c7a-kube-api-access-rs2qt\") pod \"limitador-limitador-74dfcfb894-hdchk\" (UID: \"64920c2d-f5e0-4370-95b9-fd7861185c7a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:02.113350 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.113268 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/64920c2d-f5e0-4370-95b9-fd7861185c7a-config-file\") pod \"limitador-limitador-74dfcfb894-hdchk\" (UID: \"64920c2d-f5e0-4370-95b9-fd7861185c7a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:02.142575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.142558 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xtrmf\"" Oct 31 14:50:02.142737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.142722 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 14:50:02.144797 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.144779 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hdchk"] Oct 31 14:50:02.214381 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.214355 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/64920c2d-f5e0-4370-95b9-fd7861185c7a-config-file\") pod \"limitador-limitador-74dfcfb894-hdchk\" (UID: \"64920c2d-f5e0-4370-95b9-fd7861185c7a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:02.214511 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.214402 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2qt\" (UniqueName: \"kubernetes.io/projected/64920c2d-f5e0-4370-95b9-fd7861185c7a-kube-api-access-rs2qt\") pod \"limitador-limitador-74dfcfb894-hdchk\" (UID: \"64920c2d-f5e0-4370-95b9-fd7861185c7a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:02.214973 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.214956 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/64920c2d-f5e0-4370-95b9-fd7861185c7a-config-file\") pod \"limitador-limitador-74dfcfb894-hdchk\" (UID: \"64920c2d-f5e0-4370-95b9-fd7861185c7a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:02.275587 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.275557 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2qt\" (UniqueName: \"kubernetes.io/projected/64920c2d-f5e0-4370-95b9-fd7861185c7a-kube-api-access-rs2qt\") pod \"limitador-limitador-74dfcfb894-hdchk\" (UID: \"64920c2d-f5e0-4370-95b9-fd7861185c7a\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:02.328970 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.328935 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hdchk"] Oct 31 14:50:02.399988 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.399915 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:02.772861 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.772712 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hdchk"] Oct 31 14:50:02.775774 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:50:02.775748 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64920c2d_f5e0_4370_95b9_fd7861185c7a.slice/crio-4de1340c2b2aa2bf3edea08ccc885ac20d363271af184ec9760232c13c8a7818 WatchSource:0}: Error finding container 4de1340c2b2aa2bf3edea08ccc885ac20d363271af184ec9760232c13c8a7818: Status 404 returned error can't find the container with id 4de1340c2b2aa2bf3edea08ccc885ac20d363271af184ec9760232c13c8a7818 Oct 31 14:50:02.901163 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:02.901130 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" event={"ID":"64920c2d-f5e0-4370-95b9-fd7861185c7a","Type":"ContainerStarted","Data":"4de1340c2b2aa2bf3edea08ccc885ac20d363271af184ec9760232c13c8a7818"} Oct 31 14:50:05.914026 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:05.913993 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" event={"ID":"64920c2d-f5e0-4370-95b9-fd7861185c7a","Type":"ContainerStarted","Data":"661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3"} Oct 31 14:50:05.914417 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:05.914119 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:06.096680 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.096631 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" podStartSLOduration=1.499755581 podStartE2EDuration="4.096616813s" totalImagesPullingTime="2.596861232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:50:06.027434672 +0000 UTC m=+249.238676085" watchObservedRunningTime="2026-10-31 14:50:06.096616813 +0000 UTC m=+249.307858224" Oct 31 14:50:06.097814 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.097795 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-49hxh"] Oct 31 14:50:06.101031 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.101017 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" Oct 31 14:50:06.149552 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.149525 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9qb\" (UniqueName: \"kubernetes.io/projected/47606b43-435f-4afc-9beb-788378132759-kube-api-access-td9qb\") pod \"authorino-56c6c9ddd5-49hxh\" (UID: \"47606b43-435f-4afc-9beb-788378132759\") " pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" Oct 31 14:50:06.155007 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.154986 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lr8w6\"" Oct 31 14:50:06.155177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.155160 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-49hxh"] Oct 31 14:50:06.250157 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.250130 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9qb\" (UniqueName: \"kubernetes.io/projected/47606b43-435f-4afc-9beb-788378132759-kube-api-access-td9qb\") pod \"authorino-56c6c9ddd5-49hxh\" (UID: \"47606b43-435f-4afc-9beb-788378132759\") " pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" Oct 31 14:50:06.306324 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.306294 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9qb\" (UniqueName: \"kubernetes.io/projected/47606b43-435f-4afc-9beb-788378132759-kube-api-access-td9qb\") pod \"authorino-56c6c9ddd5-49hxh\" (UID: \"47606b43-435f-4afc-9beb-788378132759\") " pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" Oct 31 14:50:06.413079 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.413044 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" Oct 31 14:50:06.783179 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.783151 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-49hxh"] Oct 31 14:50:06.786268 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:50:06.786241 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47606b43_435f_4afc_9beb_788378132759.slice/crio-a9ec0d8bc00fac2870386f101db2616fa2fc67eedac2bc230487ae981a547c96 WatchSource:0}: Error finding container a9ec0d8bc00fac2870386f101db2616fa2fc67eedac2bc230487ae981a547c96: Status 404 returned error can't find the container with id a9ec0d8bc00fac2870386f101db2616fa2fc67eedac2bc230487ae981a547c96 Oct 31 14:50:06.919231 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:06.919196 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" event={"ID":"47606b43-435f-4afc-9beb-788378132759","Type":"ContainerStarted","Data":"a9ec0d8bc00fac2870386f101db2616fa2fc67eedac2bc230487ae981a547c96"} Oct 31 14:50:09.932643 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:09.932604 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" event={"ID":"47606b43-435f-4afc-9beb-788378132759","Type":"ContainerStarted","Data":"211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455"} Oct 31 14:50:10.047556 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:10.047505 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" podStartSLOduration=1.621412829 podStartE2EDuration="4.047489282s" totalImagesPullingTime="2.426076453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:50:10.045813044 +0000 UTC m=+253.257054454" watchObservedRunningTime="2026-10-31 14:50:10.047489282 +0000 UTC m=+253.258730694" Oct 31 14:50:13.367300 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.367265 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7"] Oct 31 14:50:13.369414 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.369398 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.402423 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.402395 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdxgd\" (UniqueName: \"kubernetes.io/projected/e191e1bb-a406-4fa3-b9bf-b27502111962-kube-api-access-sdxgd\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.402547 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.402440 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e191e1bb-a406-4fa3-b9bf-b27502111962-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.402547 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.402468 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e191e1bb-a406-4fa3-b9bf-b27502111962-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.402631 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.402568 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e191e1bb-a406-4fa3-b9bf-b27502111962-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.426236 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.426212 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-m95mw\"" Oct 31 14:50:13.426363 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.426345 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 14:50:13.426438 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.426427 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 14:50:13.503543 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.503515 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdxgd\" (UniqueName: \"kubernetes.io/projected/e191e1bb-a406-4fa3-b9bf-b27502111962-kube-api-access-sdxgd\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.503685 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.503560 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e191e1bb-a406-4fa3-b9bf-b27502111962-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.503770 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.503679 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e191e1bb-a406-4fa3-b9bf-b27502111962-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.503813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.503776 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e191e1bb-a406-4fa3-b9bf-b27502111962-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.503901 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.503886 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e191e1bb-a406-4fa3-b9bf-b27502111962-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.504064 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.504048 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e191e1bb-a406-4fa3-b9bf-b27502111962-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.574193 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.574155 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 14:50:13.575663 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.575633 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7"] Oct 31 14:50:13.576803 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.576772 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdxgd\" (UniqueName: \"kubernetes.io/projected/e191e1bb-a406-4fa3-b9bf-b27502111962-kube-api-access-sdxgd\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.585218 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.585198 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e191e1bb-a406-4fa3-b9bf-b27502111962-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-2p2f7\" (UID: \"e191e1bb-a406-4fa3-b9bf-b27502111962\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:13.680409 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:13.680356 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:14.099925 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:14.099896 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7"] Oct 31 14:50:14.102903 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:50:14.102878 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode191e1bb_a406_4fa3_b9bf_b27502111962.slice/crio-4bde0f88affafc7e03d2e18ce4f1ca69cb75f6d489e832e987bff61e66996370 WatchSource:0}: Error finding container 4bde0f88affafc7e03d2e18ce4f1ca69cb75f6d489e832e987bff61e66996370: Status 404 returned error can't find the container with id 4bde0f88affafc7e03d2e18ce4f1ca69cb75f6d489e832e987bff61e66996370 Oct 31 14:50:14.953309 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:14.953276 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" event={"ID":"e191e1bb-a406-4fa3-b9bf-b27502111962","Type":"ContainerStarted","Data":"4bde0f88affafc7e03d2e18ce4f1ca69cb75f6d489e832e987bff61e66996370"} Oct 31 14:50:16.920301 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:16.920268 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:16.962861 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:16.962826 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" event={"ID":"e191e1bb-a406-4fa3-b9bf-b27502111962","Type":"ContainerStarted","Data":"dcfd845b9a6a30be65286e1bc56c211eef973f62718c3d4f2b5e7c3f73e21be3"} Oct 31 14:50:16.963070 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:16.962889 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:17.151221 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:17.151173 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" podStartSLOduration=2.35110873 podStartE2EDuration="4.151156521s" totalImagesPullingTime="1.800047791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:50:17.149789612 +0000 UTC m=+260.361031024" watchObservedRunningTime="2026-10-31 14:50:17.151156521 +0000 UTC m=+260.362397933" Oct 31 14:50:19.800729 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:19.800682 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hdchk"] Oct 31 14:50:19.801126 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:19.800887 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" podUID="64920c2d-f5e0-4370-95b9-fd7861185c7a" containerName="limitador" containerID="cri-o://661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3" gracePeriod=30 Oct 31 14:50:20.409141 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.409114 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:20.466851 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.466825 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64920c2d-f5e0-4370-95b9-fd7861185c7a-config-file\" (UniqueName: \"kubernetes.io/configmap/64920c2d-f5e0-4370-95b9-fd7861185c7a-config-file\") pod \"64920c2d-f5e0-4370-95b9-fd7861185c7a\" (UID: \"64920c2d-f5e0-4370-95b9-fd7861185c7a\") " Oct 31 14:50:20.466966 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.466869 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64920c2d-f5e0-4370-95b9-fd7861185c7a-kube-api-access-rs2qt\" (UniqueName: \"kubernetes.io/projected/64920c2d-f5e0-4370-95b9-fd7861185c7a-kube-api-access-rs2qt\") pod \"64920c2d-f5e0-4370-95b9-fd7861185c7a\" (UID: \"64920c2d-f5e0-4370-95b9-fd7861185c7a\") " Oct 31 14:50:20.467177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.467159 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64920c2d-f5e0-4370-95b9-fd7861185c7a-config-file" pod "64920c2d-f5e0-4370-95b9-fd7861185c7a" (UID: "64920c2d-f5e0-4370-95b9-fd7861185c7a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:50:20.469120 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.469095 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64920c2d-f5e0-4370-95b9-fd7861185c7a-kube-api-access-rs2qt" pod "64920c2d-f5e0-4370-95b9-fd7861185c7a" (UID: "64920c2d-f5e0-4370-95b9-fd7861185c7a"). InnerVolumeSpecName "kube-api-access-rs2qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:50:20.568251 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.568228 4399 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/64920c2d-f5e0-4370-95b9-fd7861185c7a-config-file\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:50:20.568251 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.568248 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs2qt\" (UniqueName: \"kubernetes.io/projected/64920c2d-f5e0-4370-95b9-fd7861185c7a-kube-api-access-rs2qt\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:50:20.979379 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.979345 4399 generic.go:356] "Generic (PLEG): container finished" podID="64920c2d-f5e0-4370-95b9-fd7861185c7a" containerID="661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3" exitCode=0 Oct 31 14:50:20.979798 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.979391 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" Oct 31 14:50:20.979798 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.979432 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" event={"ID":"64920c2d-f5e0-4370-95b9-fd7861185c7a","Type":"ContainerDied","Data":"661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3"} Oct 31 14:50:20.979798 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.979483 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hdchk" event={"ID":"64920c2d-f5e0-4370-95b9-fd7861185c7a","Type":"ContainerDied","Data":"4de1340c2b2aa2bf3edea08ccc885ac20d363271af184ec9760232c13c8a7818"} Oct 31 14:50:20.979798 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.979505 4399 scope.go:122] "RemoveContainer" containerID="661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3" Oct 31 14:50:20.987557 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.987540 4399 scope.go:122] "RemoveContainer" containerID="661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3" Oct 31 14:50:20.987854 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:50:20.987834 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3\": container with ID starting with 661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3 not found: ID does not exist" containerID="661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3" Oct 31 14:50:20.987922 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:20.987866 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3"} err="failed to get container status \"661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3\": rpc error: code = NotFound desc = could not find container \"661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3\": container with ID starting with 661b77d2db570b22e751d36b449e29eaddca13b238c1e0f8a38fa701dedcdab3 not found: ID does not exist" Oct 31 14:50:21.140154 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:21.140121 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hdchk"] Oct 31 14:50:21.189527 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:21.189470 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hdchk"] Oct 31 14:50:22.894977 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:22.894936 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64920c2d-f5e0-4370-95b9-fd7861185c7a" path="/var/lib/kubelet/pods/64920c2d-f5e0-4370-95b9-fd7861185c7a/volumes" Oct 31 14:50:26.205290 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:50:26.205251 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:50:27.735872 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.735835 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x"] Oct 31 14:50:27.737234 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.737206 4399 cpu_manager.go:414] "containerMap: removing container" podUID="64920c2d-f5e0-4370-95b9-fd7861185c7a" containerName="limitador" Oct 31 14:50:27.737397 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.737384 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64920c2d-f5e0-4370-95b9-fd7861185c7a" containerName="limitador" Oct 31 14:50:27.737691 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.737676 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="64920c2d-f5e0-4370-95b9-fd7861185c7a" containerName="limitador" Oct 31 14:50:27.740949 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.740921 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:27.794190 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.794158 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x"] Oct 31 14:50:27.829624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.829585 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzzwr\" (UniqueName: \"kubernetes.io/projected/8100f287-aef3-49cf-9fb4-a6363db3e4b1-kube-api-access-zzzwr\") pod \"postgres-7cdbf6dc44-7259x\" (UID: \"8100f287-aef3-49cf-9fb4-a6363db3e4b1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:27.829826 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.829684 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8100f287-aef3-49cf-9fb4-a6363db3e4b1-data\") pod \"postgres-7cdbf6dc44-7259x\" (UID: \"8100f287-aef3-49cf-9fb4-a6363db3e4b1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:27.837502 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.837483 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:50:27.841904 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.841879 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:50:27.842011 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.842003 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dtk5s\"" Oct 31 14:50:27.842766 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.842754 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 14:50:27.931157 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.931123 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzzwr\" (UniqueName: \"kubernetes.io/projected/8100f287-aef3-49cf-9fb4-a6363db3e4b1-kube-api-access-zzzwr\") pod \"postgres-7cdbf6dc44-7259x\" (UID: \"8100f287-aef3-49cf-9fb4-a6363db3e4b1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:27.931157 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.931157 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8100f287-aef3-49cf-9fb4-a6363db3e4b1-data\") pod \"postgres-7cdbf6dc44-7259x\" (UID: \"8100f287-aef3-49cf-9fb4-a6363db3e4b1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:27.931490 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.931472 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8100f287-aef3-49cf-9fb4-a6363db3e4b1-data\") pod \"postgres-7cdbf6dc44-7259x\" (UID: \"8100f287-aef3-49cf-9fb4-a6363db3e4b1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:27.987372 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:27.987300 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzzwr\" (UniqueName: \"kubernetes.io/projected/8100f287-aef3-49cf-9fb4-a6363db3e4b1-kube-api-access-zzzwr\") pod \"postgres-7cdbf6dc44-7259x\" (UID: \"8100f287-aef3-49cf-9fb4-a6363db3e4b1\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:28.052738 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:28.052691 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:28.420499 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:28.420404 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x"] Oct 31 14:50:28.423714 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:50:28.423678 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8100f287_aef3_49cf_9fb4_a6363db3e4b1.slice/crio-db1313d5fb4ce504a6052cd7b2aa6d3985c940be8264cc0babb2ba80455cc45e WatchSource:0}: Error finding container db1313d5fb4ce504a6052cd7b2aa6d3985c940be8264cc0babb2ba80455cc45e: Status 404 returned error can't find the container with id db1313d5fb4ce504a6052cd7b2aa6d3985c940be8264cc0babb2ba80455cc45e Oct 31 14:50:29.012799 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:29.012762 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" event={"ID":"8100f287-aef3-49cf-9fb4-a6363db3e4b1","Type":"ContainerStarted","Data":"db1313d5fb4ce504a6052cd7b2aa6d3985c940be8264cc0babb2ba80455cc45e"} Oct 31 14:50:36.052100 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:36.052067 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" event={"ID":"8100f287-aef3-49cf-9fb4-a6363db3e4b1","Type":"ContainerStarted","Data":"67356030da07f60f7aa1c35070ee1dccbdc1907869952682c629094c131ad89c"} Oct 31 14:50:36.052462 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:36.052150 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:36.165865 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:36.165807 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" podStartSLOduration=1.6422405900000001 podStartE2EDuration="9.165792722s" totalImagesPullingTime="7.523552132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:50:36.164531792 +0000 UTC m=+279.375773206" watchObservedRunningTime="2026-10-31 14:50:36.165792722 +0000 UTC m=+279.377034125" Oct 31 14:50:37.967390 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:37.967364 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-2p2f7" Oct 31 14:50:42.084750 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:42.084722 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7259x" Oct 31 14:50:56.205616 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:50:56.205581 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:50:59.008840 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.008803 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm"] Oct 31 14:50:59.014017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.013997 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.070883 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.070855 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm"] Oct 31 14:50:59.099160 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.099137 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f35f6cb1-6438-4d87-a0ac-078e5960373e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.099278 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.099241 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f35f6cb1-6438-4d87-a0ac-078e5960373e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.099318 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.099278 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89jfl\" (UniqueName: \"kubernetes.io/projected/f35f6cb1-6438-4d87-a0ac-078e5960373e-kube-api-access-89jfl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.099414 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.099393 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f35f6cb1-6438-4d87-a0ac-078e5960373e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.118104 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.118076 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 14:50:59.118104 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.118079 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 14:50:59.118282 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.118119 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 14:50:59.118282 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.118157 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qrk96\"" Oct 31 14:50:59.199977 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.199945 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f35f6cb1-6438-4d87-a0ac-078e5960373e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.199977 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.199978 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f35f6cb1-6438-4d87-a0ac-078e5960373e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.200186 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.200024 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f35f6cb1-6438-4d87-a0ac-078e5960373e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.200186 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.200057 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89jfl\" (UniqueName: \"kubernetes.io/projected/f35f6cb1-6438-4d87-a0ac-078e5960373e-kube-api-access-89jfl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.200186 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:50:59.200092 4399 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Oct 31 14:50:59.200186 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:50:59.200164 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f35f6cb1-6438-4d87-a0ac-078e5960373e-tls-cert podName:f35f6cb1-6438-4d87-a0ac-078e5960373e nodeName:}" failed. No retries permitted until 2026-10-31 14:50:59.700143984 +0000 UTC m=+302.911385374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/f35f6cb1-6438-4d87-a0ac-078e5960373e-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-28xqm" (UID: "f35f6cb1-6438-4d87-a0ac-078e5960373e") : secret "usage-logs-tenancy-proxy-tls" not found Oct 31 14:50:59.200343 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:50:59.200191 4399 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Oct 31 14:50:59.200343 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:50:59.200229 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f35f6cb1-6438-4d87-a0ac-078e5960373e-proxy-source podName:f35f6cb1-6438-4d87-a0ac-078e5960373e nodeName:}" failed. No retries permitted until 2026-10-31 14:50:59.700217824 +0000 UTC m=+302.911459241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/f35f6cb1-6438-4d87-a0ac-078e5960373e-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-28xqm" (UID: "f35f6cb1-6438-4d87-a0ac-078e5960373e") : configmap "usage-tenancy-proxy-source" not found Oct 31 14:50:59.200619 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.200600 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f35f6cb1-6438-4d87-a0ac-078e5960373e-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.256740 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.256688 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89jfl\" (UniqueName: \"kubernetes.io/projected/f35f6cb1-6438-4d87-a0ac-078e5960373e-kube-api-access-89jfl\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.703252 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.703216 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f35f6cb1-6438-4d87-a0ac-078e5960373e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.703468 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.703305 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f35f6cb1-6438-4d87-a0ac-078e5960373e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.704128 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.704102 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/f35f6cb1-6438-4d87-a0ac-078e5960373e-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.705746 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.705728 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f35f6cb1-6438-4d87-a0ac-078e5960373e-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-28xqm\" (UID: \"f35f6cb1-6438-4d87-a0ac-078e5960373e\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:50:59.926544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:50:59.926507 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:51:00.295611 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:00.295530 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm"] Oct 31 14:51:00.298307 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:51:00.298281 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf35f6cb1_6438_4d87_a0ac_078e5960373e.slice/crio-2d83f0ef9a43ed55437b40b2533ce5b268eb7faaa1495c17486406dc3ef50019 WatchSource:0}: Error finding container 2d83f0ef9a43ed55437b40b2533ce5b268eb7faaa1495c17486406dc3ef50019: Status 404 returned error can't find the container with id 2d83f0ef9a43ed55437b40b2533ce5b268eb7faaa1495c17486406dc3ef50019 Oct 31 14:51:01.153797 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:01.153751 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" event={"ID":"f35f6cb1-6438-4d87-a0ac-078e5960373e","Type":"ContainerStarted","Data":"2d83f0ef9a43ed55437b40b2533ce5b268eb7faaa1495c17486406dc3ef50019"} Oct 31 14:51:13.214101 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:13.214065 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" event={"ID":"f35f6cb1-6438-4d87-a0ac-078e5960373e","Type":"ContainerStarted","Data":"fa807eb35e325d55d5b01c7a93b5f5e74ab324c83446b7915dd4ee4bb66654d3"} Oct 31 14:51:13.214482 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:13.214217 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:51:13.330604 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:13.330548 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" podStartSLOduration=2.822750347 podStartE2EDuration="15.330534353s" totalImagesPullingTime="12.507784006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:13.329798998 +0000 UTC m=+316.541040410" watchObservedRunningTime="2026-10-31 14:51:13.330534353 +0000 UTC m=+316.541775764" Oct 31 14:51:19.216855 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.216818 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-knls7"] Oct 31 14:51:19.220476 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.220455 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:19.269171 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.269140 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktn6l\" (UniqueName: \"kubernetes.io/projected/698a358f-ef89-4279-a36c-dd211218127e-kube-api-access-ktn6l\") pod \"model-serving-api-75bd89fdfc-knls7\" (UID: \"698a358f-ef89-4279-a36c-dd211218127e\") " pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:19.269360 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.269213 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698a358f-ef89-4279-a36c-dd211218127e-tls-certs\") pod \"model-serving-api-75bd89fdfc-knls7\" (UID: \"698a358f-ef89-4279-a36c-dd211218127e\") " pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:19.276935 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.276904 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-knls7"] Oct 31 14:51:19.324579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.324538 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 14:51:19.324761 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.324600 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-t5rnb\"" Oct 31 14:51:19.370382 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.370352 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698a358f-ef89-4279-a36c-dd211218127e-tls-certs\") pod \"model-serving-api-75bd89fdfc-knls7\" (UID: \"698a358f-ef89-4279-a36c-dd211218127e\") " pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:19.370524 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.370398 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktn6l\" (UniqueName: \"kubernetes.io/projected/698a358f-ef89-4279-a36c-dd211218127e-kube-api-access-ktn6l\") pod \"model-serving-api-75bd89fdfc-knls7\" (UID: \"698a358f-ef89-4279-a36c-dd211218127e\") " pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:19.372996 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.372971 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/698a358f-ef89-4279-a36c-dd211218127e-tls-certs\") pod \"model-serving-api-75bd89fdfc-knls7\" (UID: \"698a358f-ef89-4279-a36c-dd211218127e\") " pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:19.427725 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.427663 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktn6l\" (UniqueName: \"kubernetes.io/projected/698a358f-ef89-4279-a36c-dd211218127e-kube-api-access-ktn6l\") pod \"model-serving-api-75bd89fdfc-knls7\" (UID: \"698a358f-ef89-4279-a36c-dd211218127e\") " pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:19.532909 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.532835 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:19.903464 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.903441 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-knls7"] Oct 31 14:51:19.906176 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:51:19.906149 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698a358f_ef89_4279_a36c_dd211218127e.slice/crio-2a420e271744770462ebf2849633ce094e294ab98cb76d7c0f266ac4c4b5f02f WatchSource:0}: Error finding container 2a420e271744770462ebf2849633ce094e294ab98cb76d7c0f266ac4c4b5f02f: Status 404 returned error can't find the container with id 2a420e271744770462ebf2849633ce094e294ab98cb76d7c0f266ac4c4b5f02f Oct 31 14:51:19.908005 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:19.907987 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:51:20.244139 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:20.244110 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-knls7" event={"ID":"698a358f-ef89-4279-a36c-dd211218127e","Type":"ContainerStarted","Data":"2a420e271744770462ebf2849633ce094e294ab98cb76d7c0f266ac4c4b5f02f"} Oct 31 14:51:22.256651 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:22.256618 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-knls7" event={"ID":"698a358f-ef89-4279-a36c-dd211218127e","Type":"ContainerStarted","Data":"6c93221218ff5339f077efc83e43d45e92d5c6846e520b0f5592cf19e1c2328c"} Oct 31 14:51:22.257006 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:22.256777 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:22.370126 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:22.370047 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-knls7" podStartSLOduration=1.176282556 podStartE2EDuration="3.370034794s" totalImagesPullingTime="2.193752238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:22.369180063 +0000 UTC m=+325.580421475" watchObservedRunningTime="2026-10-31 14:51:22.370034794 +0000 UTC m=+325.581276206" Oct 31 14:51:24.224763 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:24.224738 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-28xqm" Oct 31 14:51:26.205299 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:51:26.205264 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:51:32.727809 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.727778 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p"] Oct 31 14:51:32.731458 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.731442 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.835036 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.835007 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\"" Oct 31 14:51:32.835036 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.835025 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 14:51:32.835204 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.835092 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 14:51:32.844727 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.844693 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p"] Oct 31 14:51:32.887940 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.887918 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-657c8\" (UniqueName: \"kubernetes.io/projected/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-kube-api-access-657c8\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.888057 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.887945 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-maas-api-tls\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.888057 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.887973 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-openshift-service-ca\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.888057 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.887998 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-metrics-certs\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.989375 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.989315 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-657c8\" (UniqueName: \"kubernetes.io/projected/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-kube-api-access-657c8\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.989375 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.989346 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-maas-api-tls\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.989518 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.989383 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-openshift-service-ca\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.989518 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.989417 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-metrics-certs\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.990092 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.990070 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-openshift-service-ca\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.991891 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.991872 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-maas-api-tls\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:32.991980 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:32.991932 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-metrics-certs\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:33.046121 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:33.046090 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-657c8\" (UniqueName: \"kubernetes.io/projected/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-kube-api-access-657c8\") pod \"maas-api-654bbcd8c-h5t8p\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:33.264296 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:33.264232 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-knls7" Oct 31 14:51:33.343527 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:33.343496 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:33.715571 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:33.715542 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p"] Oct 31 14:51:33.718324 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:51:33.718301 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4b797a_4119_4fee_9d93_c0d09b7b4b24.slice/crio-91c7a01e5d1c880c1ad8014ff8493710e8e057454ef943508df54b022e0f4839 WatchSource:0}: Error finding container 91c7a01e5d1c880c1ad8014ff8493710e8e057454ef943508df54b022e0f4839: Status 404 returned error can't find the container with id 91c7a01e5d1c880c1ad8014ff8493710e8e057454ef943508df54b022e0f4839 Oct 31 14:51:34.308688 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:34.308655 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" event={"ID":"eb4b797a-4119-4fee-9d93-c0d09b7b4b24","Type":"ContainerStarted","Data":"91c7a01e5d1c880c1ad8014ff8493710e8e057454ef943508df54b022e0f4839"} Oct 31 14:51:41.339741 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:41.339682 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" event={"ID":"eb4b797a-4119-4fee-9d93-c0d09b7b4b24","Type":"ContainerStarted","Data":"b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522"} Oct 31 14:51:41.340204 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:41.339757 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:41.454486 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:41.454441 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" podStartSLOduration=2.741730353 podStartE2EDuration="9.454427012s" totalImagesPullingTime="6.712696659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:41.452858885 +0000 UTC m=+344.664100296" watchObservedRunningTime="2026-10-31 14:51:41.454427012 +0000 UTC m=+344.665668424" Oct 31 14:51:47.349776 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:47.349751 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:48.349308 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.349278 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-g59d7"] Oct 31 14:51:48.359952 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.359932 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.409618 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.409590 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-g59d7"] Oct 31 14:51:48.525000 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.524966 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqrgk\" (UniqueName: \"kubernetes.io/projected/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-kube-api-access-xqrgk\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.525000 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.525006 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-openshift-service-ca\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.525229 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.525025 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-metrics-certs\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.525229 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.525203 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-maas-api-tls\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.626578 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.626508 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqrgk\" (UniqueName: \"kubernetes.io/projected/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-kube-api-access-xqrgk\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.626578 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.626547 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-openshift-service-ca\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.626578 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.626565 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-metrics-certs\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.626861 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.626610 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-maas-api-tls\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.627248 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.627225 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-openshift-service-ca\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.629138 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.629116 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-metrics-certs\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.629138 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.629114 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-maas-api-tls\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.682857 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.682833 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqrgk\" (UniqueName: \"kubernetes.io/projected/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-kube-api-access-xqrgk\") pod \"maas-api-84fff96864-g59d7\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:48.971780 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:48.971748 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:49.343233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:49.343211 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-g59d7"] Oct 31 14:51:49.345678 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:51:49.345654 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d8aad2_88c7_46fb_b730_b6aa7a2d4c2d.slice/crio-57c881737358686a245b9c7358ea45251561d7fabe5439f3c7cf6db7fcfd74d8 WatchSource:0}: Error finding container 57c881737358686a245b9c7358ea45251561d7fabe5439f3c7cf6db7fcfd74d8: Status 404 returned error can't find the container with id 57c881737358686a245b9c7358ea45251561d7fabe5439f3c7cf6db7fcfd74d8 Oct 31 14:51:49.374760 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:49.374727 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" event={"ID":"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d","Type":"ContainerStarted","Data":"57c881737358686a245b9c7358ea45251561d7fabe5439f3c7cf6db7fcfd74d8"} Oct 31 14:51:50.380353 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:50.380252 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" event={"ID":"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d","Type":"ContainerStarted","Data":"a7130cb15d7212afe92be879b96e7a54013535f8b517cecb47f357f1f4b77442"} Oct 31 14:51:50.380766 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:50.380355 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:50.498440 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:50.498385 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" podStartSLOduration=1.7300578739999999 podStartE2EDuration="2.49837071s" totalImagesPullingTime="768.312836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:51:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:51:50.497149276 +0000 UTC m=+353.708390687" watchObservedRunningTime="2026-10-31 14:51:50.49837071 +0000 UTC m=+353.709612122" Oct 31 14:51:51.437399 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:51.437317 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-49hxh"] Oct 31 14:51:51.437816 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:51.437545 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" podUID="47606b43-435f-4afc-9beb-788378132759" containerName="authorino" containerID="cri-o://211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455" gracePeriod=30 Oct 31 14:51:51.690097 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:51.690039 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" Oct 31 14:51:51.857688 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:51.857663 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47606b43-435f-4afc-9beb-788378132759-kube-api-access-td9qb\" (UniqueName: \"kubernetes.io/projected/47606b43-435f-4afc-9beb-788378132759-kube-api-access-td9qb\") pod \"47606b43-435f-4afc-9beb-788378132759\" (UID: \"47606b43-435f-4afc-9beb-788378132759\") " Oct 31 14:51:51.859813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:51.859781 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47606b43-435f-4afc-9beb-788378132759-kube-api-access-td9qb" pod "47606b43-435f-4afc-9beb-788378132759" (UID: "47606b43-435f-4afc-9beb-788378132759"). InnerVolumeSpecName "kube-api-access-td9qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:51:51.958370 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:51.958343 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-td9qb\" (UniqueName: \"kubernetes.io/projected/47606b43-435f-4afc-9beb-788378132759-kube-api-access-td9qb\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:52.389534 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.389456 4399 generic.go:356] "Generic (PLEG): container finished" podID="47606b43-435f-4afc-9beb-788378132759" containerID="211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455" exitCode=0 Oct 31 14:51:52.389534 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.389501 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" Oct 31 14:51:52.389534 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.389517 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" event={"ID":"47606b43-435f-4afc-9beb-788378132759","Type":"ContainerDied","Data":"211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455"} Oct 31 14:51:52.389823 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.389548 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-49hxh" event={"ID":"47606b43-435f-4afc-9beb-788378132759","Type":"ContainerDied","Data":"a9ec0d8bc00fac2870386f101db2616fa2fc67eedac2bc230487ae981a547c96"} Oct 31 14:51:52.389823 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.389563 4399 scope.go:122] "RemoveContainer" containerID="211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455" Oct 31 14:51:52.398321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.398306 4399 scope.go:122] "RemoveContainer" containerID="211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455" Oct 31 14:51:52.398582 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:51:52.398562 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455\": container with ID starting with 211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455 not found: ID does not exist" containerID="211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455" Oct 31 14:51:52.398642 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.398593 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455"} err="failed to get container status \"211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455\": rpc error: code = NotFound desc = could not find container \"211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455\": container with ID starting with 211601d7bfde675e17826e361ac76cf002f23f14c8f4d9019786646c2fa8e455 not found: ID does not exist" Oct 31 14:51:52.550666 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.550642 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-49hxh"] Oct 31 14:51:52.554197 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.554176 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-49hxh"] Oct 31 14:51:52.894382 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:52.894352 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47606b43-435f-4afc-9beb-788378132759" path="/var/lib/kubelet/pods/47606b43-435f-4afc-9beb-788378132759/volumes" Oct 31 14:51:56.206174 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:51:56.206141 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:51:56.390761 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.390736 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 14:51:56.556318 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.556232 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p"] Oct 31 14:51:56.556528 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.556501 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" podUID="eb4b797a-4119-4fee-9d93-c0d09b7b4b24" containerName="maas-api" containerID="cri-o://b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522" gracePeriod=30 Oct 31 14:51:56.794260 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.794237 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:56.894255 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.894176 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-maas-api-tls\") pod \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " Oct 31 14:51:56.894255 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.894210 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-openshift-service-ca\") pod \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " Oct 31 14:51:56.894464 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.894280 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-kube-api-access-657c8\" (UniqueName: \"kubernetes.io/projected/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-kube-api-access-657c8\") pod \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " Oct 31 14:51:56.894464 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.894384 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-metrics-certs\") pod \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\" (UID: \"eb4b797a-4119-4fee-9d93-c0d09b7b4b24\") " Oct 31 14:51:56.894578 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.894513 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-openshift-service-ca" pod "eb4b797a-4119-4fee-9d93-c0d09b7b4b24" (UID: "eb4b797a-4119-4fee-9d93-c0d09b7b4b24"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:51:56.894641 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.894617 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:56.896528 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.896498 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-kube-api-access-657c8" pod "eb4b797a-4119-4fee-9d93-c0d09b7b4b24" (UID: "eb4b797a-4119-4fee-9d93-c0d09b7b4b24"). InnerVolumeSpecName "kube-api-access-657c8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:51:56.896626 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.896533 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-metrics-certs" pod "eb4b797a-4119-4fee-9d93-c0d09b7b4b24" (UID: "eb4b797a-4119-4fee-9d93-c0d09b7b4b24"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:51:56.896626 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.896604 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-maas-api-tls" pod "eb4b797a-4119-4fee-9d93-c0d09b7b4b24" (UID: "eb4b797a-4119-4fee-9d93-c0d09b7b4b24"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:51:56.995342 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.995311 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:56.995342 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.995337 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-657c8\" (UniqueName: \"kubernetes.io/projected/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-kube-api-access-657c8\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:56.995342 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:56.995347 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb4b797a-4119-4fee-9d93-c0d09b7b4b24-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:51:57.412749 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:57.412712 4399 generic.go:356] "Generic (PLEG): container finished" podID="eb4b797a-4119-4fee-9d93-c0d09b7b4b24" containerID="b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522" exitCode=0 Oct 31 14:51:57.413190 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:57.412758 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" Oct 31 14:51:57.413190 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:57.412776 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" event={"ID":"eb4b797a-4119-4fee-9d93-c0d09b7b4b24","Type":"ContainerDied","Data":"b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522"} Oct 31 14:51:57.413190 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:57.412824 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p" event={"ID":"eb4b797a-4119-4fee-9d93-c0d09b7b4b24","Type":"ContainerDied","Data":"91c7a01e5d1c880c1ad8014ff8493710e8e057454ef943508df54b022e0f4839"} Oct 31 14:51:57.413190 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:57.412845 4399 scope.go:122] "RemoveContainer" containerID="b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522" Oct 31 14:51:57.421404 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:57.421386 4399 scope.go:122] "RemoveContainer" containerID="b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522" Oct 31 14:51:57.421656 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:51:57.421638 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522\": container with ID starting with b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522 not found: ID does not exist" containerID="b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522" Oct 31 14:51:57.421740 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:57.421664 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522"} err="failed to get container status \"b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522\": rpc error: code = NotFound desc = could not find container \"b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522\": container with ID starting with b939e6be686b64348d948b11274128f5254ab4bd28689206ac95492b2f6a8522 not found: ID does not exist" Oct 31 14:51:57.575960 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:57.575927 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p"] Oct 31 14:51:57.624652 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:57.624623 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-654bbcd8c-h5t8p"] Oct 31 14:51:58.894220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:51:58.894193 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb4b797a-4119-4fee-9d93-c0d09b7b4b24" path="/var/lib/kubelet/pods/eb4b797a-4119-4fee-9d93-c0d09b7b4b24/volumes" Oct 31 14:52:26.206250 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:52:26.206215 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:52:56.209054 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:52:56.209014 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:53:08.508836 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.508803 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lpsbj"] Oct 31 14:53:08.509233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.509173 4399 cpu_manager.go:414] "containerMap: removing container" podUID="eb4b797a-4119-4fee-9d93-c0d09b7b4b24" containerName="maas-api" Oct 31 14:53:08.509233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.509184 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb4b797a-4119-4fee-9d93-c0d09b7b4b24" containerName="maas-api" Oct 31 14:53:08.509233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.509210 4399 cpu_manager.go:414] "containerMap: removing container" podUID="47606b43-435f-4afc-9beb-788378132759" containerName="authorino" Oct 31 14:53:08.509233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.509215 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47606b43-435f-4afc-9beb-788378132759" containerName="authorino" Oct 31 14:53:08.509375 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.509280 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb4b797a-4119-4fee-9d93-c0d09b7b4b24" containerName="maas-api" Oct 31 14:53:08.509375 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.509297 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="47606b43-435f-4afc-9beb-788378132759" containerName="authorino" Oct 31 14:53:08.512303 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.512284 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:08.566300 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.566276 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qvj\" (UniqueName: \"kubernetes.io/projected/80b13e8d-389c-44f1-82df-cbb312504cd5-kube-api-access-88qvj\") pod \"ai-gateway-controller-7d85675f98-lpsbj\" (UID: \"80b13e8d-389c-44f1-82df-cbb312504cd5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:08.566447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.566331 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b13e8d-389c-44f1-82df-cbb312504cd5-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-lpsbj\" (UID: \"80b13e8d-389c-44f1-82df-cbb312504cd5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:08.567342 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.567323 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lpsbj"] Oct 31 14:53:08.602107 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.602084 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 14:53:08.615265 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.615243 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-z4lnh\"" Oct 31 14:53:08.615265 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.615250 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 14:53:08.666798 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.666771 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qvj\" (UniqueName: \"kubernetes.io/projected/80b13e8d-389c-44f1-82df-cbb312504cd5-kube-api-access-88qvj\") pod \"ai-gateway-controller-7d85675f98-lpsbj\" (UID: \"80b13e8d-389c-44f1-82df-cbb312504cd5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:08.666951 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.666808 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b13e8d-389c-44f1-82df-cbb312504cd5-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-lpsbj\" (UID: \"80b13e8d-389c-44f1-82df-cbb312504cd5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:08.669170 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.669148 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b13e8d-389c-44f1-82df-cbb312504cd5-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-lpsbj\" (UID: \"80b13e8d-389c-44f1-82df-cbb312504cd5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:08.723177 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.723147 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qvj\" (UniqueName: \"kubernetes.io/projected/80b13e8d-389c-44f1-82df-cbb312504cd5-kube-api-access-88qvj\") pod \"ai-gateway-controller-7d85675f98-lpsbj\" (UID: \"80b13e8d-389c-44f1-82df-cbb312504cd5\") " pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:08.824857 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:08.824801 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:09.192431 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:09.192363 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lpsbj"] Oct 31 14:53:09.195480 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:53:09.195453 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b13e8d_389c_44f1_82df_cbb312504cd5.slice/crio-108599541cdea9d24f197357ecbb1885e3afcb8af1fc7e90832690a0bcd4787c WatchSource:0}: Error finding container 108599541cdea9d24f197357ecbb1885e3afcb8af1fc7e90832690a0bcd4787c: Status 404 returned error can't find the container with id 108599541cdea9d24f197357ecbb1885e3afcb8af1fc7e90832690a0bcd4787c Oct 31 14:53:09.727536 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:09.727501 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" event={"ID":"80b13e8d-389c-44f1-82df-cbb312504cd5","Type":"ContainerStarted","Data":"108599541cdea9d24f197357ecbb1885e3afcb8af1fc7e90832690a0bcd4787c"} Oct 31 14:53:12.742664 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:12.742626 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" event={"ID":"80b13e8d-389c-44f1-82df-cbb312504cd5","Type":"ContainerStarted","Data":"b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8"} Oct 31 14:53:12.743099 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:12.742736 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:12.855771 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:12.855682 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" podStartSLOduration=2.2688065379999998 podStartE2EDuration="4.855663826s" totalImagesPullingTime="2.586857288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:53:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:53:12.854765964 +0000 UTC m=+436.066007373" watchObservedRunningTime="2026-10-31 14:53:12.855663826 +0000 UTC m=+436.066905231" Oct 31 14:53:14.895398 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:14.895369 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-pth7n"] Oct 31 14:53:14.898840 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:14.898825 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:14.912193 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:14.912172 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a05f09f-b51e-4f82-b934-7a697b365d80-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-pth7n\" (UID: \"8a05f09f-b51e-4f82-b934-7a697b365d80\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:14.912318 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:14.912236 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6g64\" (UniqueName: \"kubernetes.io/projected/8a05f09f-b51e-4f82-b934-7a697b365d80-kube-api-access-t6g64\") pod \"payload-pre-processing-7b98bb678c-pth7n\" (UID: \"8a05f09f-b51e-4f82-b934-7a697b365d80\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:14.955074 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:14.955042 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 14:53:15.004753 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.004691 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-l4msp\"" Oct 31 14:53:15.006252 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.006227 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-pth7n"] Oct 31 14:53:15.013224 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.013200 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6g64\" (UniqueName: \"kubernetes.io/projected/8a05f09f-b51e-4f82-b934-7a697b365d80-kube-api-access-t6g64\") pod \"payload-pre-processing-7b98bb678c-pth7n\" (UID: \"8a05f09f-b51e-4f82-b934-7a697b365d80\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:15.013341 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.013283 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a05f09f-b51e-4f82-b934-7a697b365d80-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-pth7n\" (UID: \"8a05f09f-b51e-4f82-b934-7a697b365d80\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:15.013990 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.013967 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a05f09f-b51e-4f82-b934-7a697b365d80-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-pth7n\" (UID: \"8a05f09f-b51e-4f82-b934-7a697b365d80\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:15.017681 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.017659 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-kzfcq"] Oct 31 14:53:15.021143 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.021126 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:15.069861 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.069831 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6g64\" (UniqueName: \"kubernetes.io/projected/8a05f09f-b51e-4f82-b934-7a697b365d80-kube-api-access-t6g64\") pod \"payload-pre-processing-7b98bb678c-pth7n\" (UID: \"8a05f09f-b51e-4f82-b934-7a697b365d80\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:15.078407 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.078384 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-kzfcq"] Oct 31 14:53:15.114676 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.114642 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aac9441-7f91-49f3-9472-8fcd34fec615-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-kzfcq\" (UID: \"5aac9441-7f91-49f3-9472-8fcd34fec615\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:15.114807 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.114683 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2f6j\" (UniqueName: \"kubernetes.io/projected/5aac9441-7f91-49f3-9472-8fcd34fec615-kube-api-access-k2f6j\") pod \"payload-processing-55fbbbf9dc-kzfcq\" (UID: \"5aac9441-7f91-49f3-9472-8fcd34fec615\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:15.212274 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.212249 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:15.215305 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.215177 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aac9441-7f91-49f3-9472-8fcd34fec615-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-kzfcq\" (UID: \"5aac9441-7f91-49f3-9472-8fcd34fec615\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:15.215305 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.215204 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2f6j\" (UniqueName: \"kubernetes.io/projected/5aac9441-7f91-49f3-9472-8fcd34fec615-kube-api-access-k2f6j\") pod \"payload-processing-55fbbbf9dc-kzfcq\" (UID: \"5aac9441-7f91-49f3-9472-8fcd34fec615\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:15.215821 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.215803 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aac9441-7f91-49f3-9472-8fcd34fec615-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-kzfcq\" (UID: \"5aac9441-7f91-49f3-9472-8fcd34fec615\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:15.271845 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.271813 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2f6j\" (UniqueName: \"kubernetes.io/projected/5aac9441-7f91-49f3-9472-8fcd34fec615-kube-api-access-k2f6j\") pod \"payload-processing-55fbbbf9dc-kzfcq\" (UID: \"5aac9441-7f91-49f3-9472-8fcd34fec615\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:15.333811 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.333774 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:15.582189 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.582166 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-pth7n"] Oct 31 14:53:15.584450 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:53:15.584420 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a05f09f_b51e_4f82_b934_7a697b365d80.slice/crio-b8874d02bf2e30f60270788cc4f1955307d32b8d80aa74341ac733f0d7f6205f WatchSource:0}: Error finding container b8874d02bf2e30f60270788cc4f1955307d32b8d80aa74341ac733f0d7f6205f: Status 404 returned error can't find the container with id b8874d02bf2e30f60270788cc4f1955307d32b8d80aa74341ac733f0d7f6205f Oct 31 14:53:15.704390 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.704356 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-kzfcq"] Oct 31 14:53:15.707363 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:53:15.707338 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aac9441_7f91_49f3_9472_8fcd34fec615.slice/crio-a88356f09eb4b0bdf2b4553d2ccfbcad1bc9823ea5e61ed953350eaa2a0e70e0 WatchSource:0}: Error finding container a88356f09eb4b0bdf2b4553d2ccfbcad1bc9823ea5e61ed953350eaa2a0e70e0: Status 404 returned error can't find the container with id a88356f09eb4b0bdf2b4553d2ccfbcad1bc9823ea5e61ed953350eaa2a0e70e0 Oct 31 14:53:15.755541 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.755504 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" event={"ID":"5aac9441-7f91-49f3-9472-8fcd34fec615","Type":"ContainerStarted","Data":"a88356f09eb4b0bdf2b4553d2ccfbcad1bc9823ea5e61ed953350eaa2a0e70e0"} Oct 31 14:53:15.757060 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:15.757036 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" event={"ID":"8a05f09f-b51e-4f82-b934-7a697b365d80","Type":"ContainerStarted","Data":"b8874d02bf2e30f60270788cc4f1955307d32b8d80aa74341ac733f0d7f6205f"} Oct 31 14:53:19.777734 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:19.777678 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" event={"ID":"5aac9441-7f91-49f3-9472-8fcd34fec615","Type":"ContainerStarted","Data":"7199a2c36b9c460cce12c8aad826a027ef465736be15d4854b65fece7e5038f8"} Oct 31 14:53:19.778225 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:19.777802 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:19.779347 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:19.779325 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" event={"ID":"8a05f09f-b51e-4f82-b934-7a697b365d80","Type":"ContainerStarted","Data":"d575509ca9f12bf0c80e8a0f45d0822d74d8269f8c78792ab710e2f48d15b7be"} Oct 31 14:53:19.779461 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:19.779450 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:19.890424 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:19.890361 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" podStartSLOduration=2.8223305119999997 podStartE2EDuration="5.890347852s" totalImagesPullingTime="3.06801734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:53:19.889504032 +0000 UTC m=+443.100745446" watchObservedRunningTime="2026-10-31 14:53:19.890347852 +0000 UTC m=+443.101589264" Oct 31 14:53:20.003807 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:20.003753 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" podStartSLOduration=2.78146131 podStartE2EDuration="6.003737718s" totalImagesPullingTime="3.222276408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:53:20.002081325 +0000 UTC m=+443.213322736" watchObservedRunningTime="2026-10-31 14:53:20.003737718 +0000 UTC m=+443.214979129" Oct 31 14:53:23.749016 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:23.748982 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:24.093596 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.093511 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-78f488d89-cv2qm"] Oct 31 14:53:24.097186 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.097164 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:24.152872 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.152840 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-78f488d89-cv2qm"] Oct 31 14:53:24.191447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.191421 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb8xj\" (UniqueName: \"kubernetes.io/projected/bd10b679-de31-4629-8256-cf75f52e76c4-kube-api-access-tb8xj\") pod \"ai-gateway-controller-78f488d89-cv2qm\" (UID: \"bd10b679-de31-4629-8256-cf75f52e76c4\") " pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:24.191575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.191453 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd10b679-de31-4629-8256-cf75f52e76c4-metrics-certs\") pod \"ai-gateway-controller-78f488d89-cv2qm\" (UID: \"bd10b679-de31-4629-8256-cf75f52e76c4\") " pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:24.292449 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.292412 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb8xj\" (UniqueName: \"kubernetes.io/projected/bd10b679-de31-4629-8256-cf75f52e76c4-kube-api-access-tb8xj\") pod \"ai-gateway-controller-78f488d89-cv2qm\" (UID: \"bd10b679-de31-4629-8256-cf75f52e76c4\") " pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:24.292449 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.292447 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd10b679-de31-4629-8256-cf75f52e76c4-metrics-certs\") pod \"ai-gateway-controller-78f488d89-cv2qm\" (UID: \"bd10b679-de31-4629-8256-cf75f52e76c4\") " pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:24.295331 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.295310 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd10b679-de31-4629-8256-cf75f52e76c4-metrics-certs\") pod \"ai-gateway-controller-78f488d89-cv2qm\" (UID: \"bd10b679-de31-4629-8256-cf75f52e76c4\") " pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:24.349084 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.349025 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb8xj\" (UniqueName: \"kubernetes.io/projected/bd10b679-de31-4629-8256-cf75f52e76c4-kube-api-access-tb8xj\") pod \"ai-gateway-controller-78f488d89-cv2qm\" (UID: \"bd10b679-de31-4629-8256-cf75f52e76c4\") " pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:24.409677 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.409633 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:24.779924 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.779899 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-78f488d89-cv2qm"] Oct 31 14:53:24.785508 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:53:24.785463 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd10b679_de31_4629_8256_cf75f52e76c4.slice/crio-5e205098e80ad0238d5e8d992e457f403b87937cd2dcca06e8186c2524b42798 WatchSource:0}: Error finding container 5e205098e80ad0238d5e8d992e457f403b87937cd2dcca06e8186c2524b42798: Status 404 returned error can't find the container with id 5e205098e80ad0238d5e8d992e457f403b87937cd2dcca06e8186c2524b42798 Oct 31 14:53:24.802309 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:24.802281 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" event={"ID":"bd10b679-de31-4629-8256-cf75f52e76c4","Type":"ContainerStarted","Data":"5e205098e80ad0238d5e8d992e457f403b87937cd2dcca06e8186c2524b42798"} Oct 31 14:53:25.808447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:25.808412 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" event={"ID":"bd10b679-de31-4629-8256-cf75f52e76c4","Type":"ContainerStarted","Data":"ab617043ec6d82e2a3cc7d78beefc6eba6f14581e47631a7ce7d875beef182a6"} Oct 31 14:53:25.808932 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:25.808488 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:25.924009 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:25.923924 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" podStartSLOduration=1.1183391089999999 podStartE2EDuration="1.923909911s" totalImagesPullingTime="805.570802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:53:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:53:25.922332404 +0000 UTC m=+449.133573815" watchObservedRunningTime="2026-10-31 14:53:25.923909911 +0000 UTC m=+449.135151322" Oct 31 14:53:26.205157 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:53:26.205121 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:53:30.787793 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:30.787764 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 14:53:30.788204 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:30.787833 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 14:53:36.815082 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:36.815049 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 14:53:36.979072 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:36.979043 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lpsbj"] Oct 31 14:53:36.979296 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:36.979259 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" podUID="80b13e8d-389c-44f1-82df-cbb312504cd5" containerName="manager" containerID="cri-o://b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8" gracePeriod=30 Oct 31 14:53:37.230269 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.230243 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:37.304044 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.304018 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80b13e8d-389c-44f1-82df-cbb312504cd5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b13e8d-389c-44f1-82df-cbb312504cd5-metrics-certs\") pod \"80b13e8d-389c-44f1-82df-cbb312504cd5\" (UID: \"80b13e8d-389c-44f1-82df-cbb312504cd5\") " Oct 31 14:53:37.304168 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.304059 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b13e8d-389c-44f1-82df-cbb312504cd5-kube-api-access-88qvj\" (UniqueName: \"kubernetes.io/projected/80b13e8d-389c-44f1-82df-cbb312504cd5-kube-api-access-88qvj\") pod \"80b13e8d-389c-44f1-82df-cbb312504cd5\" (UID: \"80b13e8d-389c-44f1-82df-cbb312504cd5\") " Oct 31 14:53:37.306292 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.306266 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b13e8d-389c-44f1-82df-cbb312504cd5-kube-api-access-88qvj" pod "80b13e8d-389c-44f1-82df-cbb312504cd5" (UID: "80b13e8d-389c-44f1-82df-cbb312504cd5"). InnerVolumeSpecName "kube-api-access-88qvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:53:37.306412 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.306355 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80b13e8d-389c-44f1-82df-cbb312504cd5-metrics-certs" pod "80b13e8d-389c-44f1-82df-cbb312504cd5" (UID: "80b13e8d-389c-44f1-82df-cbb312504cd5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:53:37.405782 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.405681 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80b13e8d-389c-44f1-82df-cbb312504cd5-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:53:37.405782 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.405739 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qvj\" (UniqueName: \"kubernetes.io/projected/80b13e8d-389c-44f1-82df-cbb312504cd5-kube-api-access-88qvj\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:53:37.863501 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.863471 4399 generic.go:356] "Generic (PLEG): container finished" podID="80b13e8d-389c-44f1-82df-cbb312504cd5" containerID="b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8" exitCode=0 Oct 31 14:53:37.863930 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.863517 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" Oct 31 14:53:37.863930 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.863556 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" event={"ID":"80b13e8d-389c-44f1-82df-cbb312504cd5","Type":"ContainerDied","Data":"b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8"} Oct 31 14:53:37.863930 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.863606 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-lpsbj" event={"ID":"80b13e8d-389c-44f1-82df-cbb312504cd5","Type":"ContainerDied","Data":"108599541cdea9d24f197357ecbb1885e3afcb8af1fc7e90832690a0bcd4787c"} Oct 31 14:53:37.863930 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.863628 4399 scope.go:122] "RemoveContainer" containerID="b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8" Oct 31 14:53:37.872231 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.872213 4399 scope.go:122] "RemoveContainer" containerID="b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8" Oct 31 14:53:37.872510 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:53:37.872488 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8\": container with ID starting with b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8 not found: ID does not exist" containerID="b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8" Oct 31 14:53:37.872589 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:37.872516 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8"} err="failed to get container status \"b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8\": rpc error: code = NotFound desc = could not find container \"b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8\": container with ID starting with b4a03b2fd61f07d7128031eae62a53432d9ce486b12369d06990e0288754f3c8 not found: ID does not exist" Oct 31 14:53:38.025688 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:38.025661 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lpsbj"] Oct 31 14:53:38.075263 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:38.075236 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-lpsbj"] Oct 31 14:53:38.893853 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:38.893819 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b13e8d-389c-44f1-82df-cbb312504cd5" path="/var/lib/kubelet/pods/80b13e8d-389c-44f1-82df-cbb312504cd5/volumes" Oct 31 14:53:46.442541 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.442514 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc"] Oct 31 14:53:46.442963 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.442942 4399 cpu_manager.go:414] "containerMap: removing container" podUID="80b13e8d-389c-44f1-82df-cbb312504cd5" containerName="manager" Oct 31 14:53:46.442963 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.442954 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b13e8d-389c-44f1-82df-cbb312504cd5" containerName="manager" Oct 31 14:53:46.443039 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.443029 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b13e8d-389c-44f1-82df-cbb312504cd5" containerName="manager" Oct 31 14:53:46.448189 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.448171 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.524472 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.524442 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc"] Oct 31 14:53:46.550249 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.550220 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 14:53:46.550249 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.550233 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m92fm\"" Oct 31 14:53:46.550249 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.550233 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 14:53:46.550435 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.550250 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 14:53:46.593294 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.593269 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.593412 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.593300 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.593412 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.593316 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.593412 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.593344 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.593576 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.593515 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55db056-9324-4750-b1fb-49c4b40c3e92-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.593618 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.593603 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5h2k\" (UniqueName: \"kubernetes.io/projected/c55db056-9324-4750-b1fb-49c4b40c3e92-kube-api-access-f5h2k\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.694659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.694593 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5h2k\" (UniqueName: \"kubernetes.io/projected/c55db056-9324-4750-b1fb-49c4b40c3e92-kube-api-access-f5h2k\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.694659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.694641 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.694852 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.694666 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.694852 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.694693 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.694852 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.694751 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.694852 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.694789 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55db056-9324-4750-b1fb-49c4b40c3e92-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.695116 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.695094 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.695183 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.695120 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.695237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.695211 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.697069 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.697050 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c55db056-9324-4750-b1fb-49c4b40c3e92-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.697324 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.697302 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c55db056-9324-4750-b1fb-49c4b40c3e92-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.753040 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.753017 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5h2k\" (UniqueName: \"kubernetes.io/projected/c55db056-9324-4750-b1fb-49c4b40c3e92-kube-api-access-f5h2k\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc\" (UID: \"c55db056-9324-4750-b1fb-49c4b40c3e92\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:46.761853 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:46.761831 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:47.131431 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:47.131410 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc"] Oct 31 14:53:47.133970 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:53:47.133933 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55db056_9324_4750_b1fb_49c4b40c3e92.slice/crio-905be80a586b49209b4a09fd17afb433e1a1fcb028c785d015c14d6351f85a83 WatchSource:0}: Error finding container 905be80a586b49209b4a09fd17afb433e1a1fcb028c785d015c14d6351f85a83: Status 404 returned error can't find the container with id 905be80a586b49209b4a09fd17afb433e1a1fcb028c785d015c14d6351f85a83 Oct 31 14:53:47.909358 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:47.909325 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" event={"ID":"c55db056-9324-4750-b1fb-49c4b40c3e92","Type":"ContainerStarted","Data":"905be80a586b49209b4a09fd17afb433e1a1fcb028c785d015c14d6351f85a83"} Oct 31 14:53:49.918010 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:49.917978 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" event={"ID":"c55db056-9324-4750-b1fb-49c4b40c3e92","Type":"ContainerStarted","Data":"ec6d2cb0d49115f05b0d81923a595fe471dacf148e3372d7869e9e64f4e65c67"} Oct 31 14:53:50.032954 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:50.032904 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" podStartSLOduration=1.808840473 podStartE2EDuration="4.032889334s" totalImagesPullingTime="2.224048861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:53:50.031282644 +0000 UTC m=+473.242524069" watchObservedRunningTime="2026-10-31 14:53:50.032889334 +0000 UTC m=+473.244130746" Oct 31 14:53:50.762731 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:50.762684 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:55.851562 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.851525 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g"] Oct 31 14:53:55.855399 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.855376 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:55.959540 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.959507 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 14:53:55.961206 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.961186 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g"] Oct 31 14:53:55.971743 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.971679 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:55.971868 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.971848 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:55.972011 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.971986 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:55.972078 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.972030 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6f4e52-0d41-4213-941c-2b63c79cb2db-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:55.972132 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.972081 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:55.972132 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:55.972118 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6c8x\" (UniqueName: \"kubernetes.io/projected/5d6f4e52-0d41-4213-941c-2b63c79cb2db-kube-api-access-n6c8x\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.073010 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.072983 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.073010 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.073012 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6f4e52-0d41-4213-941c-2b63c79cb2db-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.073194 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.073034 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.073194 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.073055 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6c8x\" (UniqueName: \"kubernetes.io/projected/5d6f4e52-0d41-4213-941c-2b63c79cb2db-kube-api-access-n6c8x\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.073194 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.073085 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.073194 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.073148 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.073453 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.073430 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.073453 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.073447 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.073575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.073520 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.075491 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.075472 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d6f4e52-0d41-4213-941c-2b63c79cb2db-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.075785 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.075765 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d6f4e52-0d41-4213-941c-2b63c79cb2db-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.130133 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.130080 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6c8x\" (UniqueName: \"kubernetes.io/projected/5d6f4e52-0d41-4213-941c-2b63c79cb2db-kube-api-access-n6c8x\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-rm55g\" (UID: \"5d6f4e52-0d41-4213-941c-2b63c79cb2db\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.167813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.167786 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:53:56.207187 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:53:56.207148 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:53:56.541282 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.541127 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g"] Oct 31 14:53:56.544025 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:53:56.543999 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6f4e52_0d41_4213_941c_2b63c79cb2db.slice/crio-fe8d4fadc678da9eaf3472580e100ea361bcb275e7aaf449ef214fdd885970e0 WatchSource:0}: Error finding container fe8d4fadc678da9eaf3472580e100ea361bcb275e7aaf449ef214fdd885970e0: Status 404 returned error can't find the container with id fe8d4fadc678da9eaf3472580e100ea361bcb275e7aaf449ef214fdd885970e0 Oct 31 14:53:56.762801 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.762776 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:56.775446 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.775426 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:56.956579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.956549 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" event={"ID":"5d6f4e52-0d41-4213-941c-2b63c79cb2db","Type":"ContainerStarted","Data":"fe8d4fadc678da9eaf3472580e100ea361bcb275e7aaf449ef214fdd885970e0"} Oct 31 14:53:56.967664 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:56.967644 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc" Oct 31 14:53:57.963002 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:57.962966 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" event={"ID":"5d6f4e52-0d41-4213-941c-2b63c79cb2db","Type":"ContainerStarted","Data":"dada0dfd7473a441b6fcf2a8a1a98988822ce5c1cd9e4f347aeb827d57545c41"} Oct 31 14:53:58.078357 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:58.078306 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" podStartSLOduration=2.504549093 podStartE2EDuration="3.078290098s" totalImagesPullingTime="573.741005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:53:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:53:58.075909999 +0000 UTC m=+481.287151412" watchObservedRunningTime="2026-10-31 14:53:58.078290098 +0000 UTC m=+481.289531510" Oct 31 14:53:58.167885 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:53:58.167859 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:54:02.937910 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:02.937875 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98"] Oct 31 14:54:02.942919 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:02.942889 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.000196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.000164 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98"] Oct 31 14:54:03.034747 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.034720 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.034914 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.034753 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.035069 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.035046 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.035147 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.035100 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.035209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.035176 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbx29\" (UniqueName: \"kubernetes.io/projected/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-kube-api-access-jbx29\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.035209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.035199 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.046944 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.046922 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 14:54:03.136216 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.136185 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.136324 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.136219 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.136324 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.136245 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbx29\" (UniqueName: \"kubernetes.io/projected/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-kube-api-access-jbx29\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.136324 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.136265 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.136324 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.136287 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.136324 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.136314 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.136680 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.136657 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.136803 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.136734 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.136803 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.136756 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.138507 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.138486 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.138712 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.138685 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.192918 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.192866 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbx29\" (UniqueName: \"kubernetes.io/projected/dea35ba4-3d75-485a-9fd0-c91e74f8f62b-kube-api-access-jbx29\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98\" (UID: \"dea35ba4-3d75-485a-9fd0-c91e74f8f62b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.257032 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.257007 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:03.630066 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.630003 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98"] Oct 31 14:54:03.632502 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:54:03.632469 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea35ba4_3d75_485a_9fd0_c91e74f8f62b.slice/crio-77f30de6f662a581fe4d1d83eb4b10b746bbd26c135f133ac1b9877e0a4f8adb WatchSource:0}: Error finding container 77f30de6f662a581fe4d1d83eb4b10b746bbd26c135f133ac1b9877e0a4f8adb: Status 404 returned error can't find the container with id 77f30de6f662a581fe4d1d83eb4b10b746bbd26c135f133ac1b9877e0a4f8adb Oct 31 14:54:03.990674 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:03.990641 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" event={"ID":"dea35ba4-3d75-485a-9fd0-c91e74f8f62b","Type":"ContainerStarted","Data":"77f30de6f662a581fe4d1d83eb4b10b746bbd26c135f133ac1b9877e0a4f8adb"} Oct 31 14:54:04.996515 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:04.996475 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" event={"ID":"dea35ba4-3d75-485a-9fd0-c91e74f8f62b","Type":"ContainerStarted","Data":"0c2d223f748ae28edb5581de66950db0287b2fdd1298b4c63b065e9e16f7d97d"} Oct 31 14:54:05.114306 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:05.114254 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" podStartSLOduration=2.533510533 podStartE2EDuration="3.114241085s" totalImagesPullingTime="580.730552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:05.112114021 +0000 UTC m=+488.323355458" watchObservedRunningTime="2026-10-31 14:54:05.114241085 +0000 UTC m=+488.325482498" Oct 31 14:54:05.257582 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:05.257500 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:06.168767 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:06.168734 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:54:06.181842 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:06.181819 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:54:07.016060 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.016033 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-rm55g" Oct 31 14:54:07.582981 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.582945 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr"] Oct 31 14:54:07.586650 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.586627 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.638580 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.638553 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 14:54:07.676000 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.675972 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.676125 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.676005 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.676125 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.676026 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996ff9-8abf-41eb-98f4-32de705fc1bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.676125 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.676042 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.676244 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.676134 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.676244 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.676159 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.676244 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.676176 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pg6l\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-kube-api-access-6pg6l\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.676376 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.676336 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.676427 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.676392 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.687963 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.687912 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" Oct 31 14:54:07.689985 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.689963 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr"] Oct 31 14:54:07.777412 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777387 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.777542 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777438 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.777542 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777455 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.777542 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777483 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996ff9-8abf-41eb-98f4-32de705fc1bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.777542 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777510 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.777770 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777557 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.777770 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777577 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.777770 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777599 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pg6l\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-kube-api-access-6pg6l\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.777770 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777681 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.777989 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.777872 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.778084 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.778062 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.778191 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.778166 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.778647 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.778623 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996ff9-8abf-41eb-98f4-32de705fc1bf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.778941 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.778917 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.780164 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.780144 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.780537 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.780516 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.835784 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.835722 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.835865 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.835791 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pg6l\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-kube-api-access-6pg6l\") pod \"maas-default-gateway-openshift-default-67bf986f99-bcmwr\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:07.899216 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:07.899191 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:08.342549 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:08.342520 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr"] Oct 31 14:54:08.345719 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:54:08.345669 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80996ff9_8abf_41eb_98f4_32de705fc1bf.slice/crio-d7b157b72efffd7c5286ebbf24a042e1d0c2aaa068fe05ba0d76ef7bdb89ebad WatchSource:0}: Error finding container d7b157b72efffd7c5286ebbf24a042e1d0c2aaa068fe05ba0d76ef7bdb89ebad: Status 404 returned error can't find the container with id d7b157b72efffd7c5286ebbf24a042e1d0c2aaa068fe05ba0d76ef7bdb89ebad Oct 31 14:54:09.014315 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:09.014281 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" event={"ID":"80996ff9-8abf-41eb-98f4-32de705fc1bf","Type":"ContainerStarted","Data":"d7b157b72efffd7c5286ebbf24a042e1d0c2aaa068fe05ba0d76ef7bdb89ebad"} Oct 31 14:54:11.788056 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.788024 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:54:11.788446 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.788100 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:54:11.788446 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.788160 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:54:11.808891 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.808863 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb"] Oct 31 14:54:11.813890 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.813873 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:11.890041 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.890014 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb"] Oct 31 14:54:11.890241 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.890221 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 14:54:11.910811 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.910785 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:11.910937 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.910815 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:11.910937 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.910836 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:11.911058 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.910969 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:11.911058 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.911038 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:11.911155 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:11.911096 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5p9s\" (UniqueName: \"kubernetes.io/projected/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-kube-api-access-p5p9s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.011874 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.011842 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.012026 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.011887 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5p9s\" (UniqueName: \"kubernetes.io/projected/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-kube-api-access-p5p9s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.012026 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.011942 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.012026 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.011959 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.012026 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.011986 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.012239 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.012049 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.012335 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.012304 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.012399 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.012361 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.012491 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.012472 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.014366 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.014343 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.014591 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.014575 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.033588 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.033561 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" event={"ID":"80996ff9-8abf-41eb-98f4-32de705fc1bf","Type":"ContainerStarted","Data":"a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca"} Oct 31 14:54:12.071423 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.071370 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5p9s\" (UniqueName: \"kubernetes.io/projected/33d1c606-bde8-4f26-9ebb-c51a6a7ab576-kube-api-access-p5p9s\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb\" (UID: \"33d1c606-bde8-4f26-9ebb-c51a6a7ab576\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.142684 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.142654 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:12.150520 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.150473 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podStartSLOduration=1.712176515 podStartE2EDuration="5.15045903s" totalImagesPullingTime="3.438282515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:12.148858235 +0000 UTC m=+495.360099647" watchObservedRunningTime="2026-10-31 14:54:12.15045903 +0000 UTC m=+495.361700441" Oct 31 14:54:12.516517 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.516489 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb"] Oct 31 14:54:12.519224 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:54:12.519199 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d1c606_bde8_4f26_9ebb_c51a6a7ab576.slice/crio-8314d8ea2cbd8a9a129c09f3f7102be792ef360247b26d1881eaf54cd3dccd97 WatchSource:0}: Error finding container 8314d8ea2cbd8a9a129c09f3f7102be792ef360247b26d1881eaf54cd3dccd97: Status 404 returned error can't find the container with id 8314d8ea2cbd8a9a129c09f3f7102be792ef360247b26d1881eaf54cd3dccd97 Oct 31 14:54:12.901636 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.901528 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 14:54:12.901636 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.901598 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 14:54:12.903215 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:12.903194 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:13.039265 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:13.039231 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" event={"ID":"33d1c606-bde8-4f26-9ebb-c51a6a7ab576","Type":"ContainerStarted","Data":"8314d8ea2cbd8a9a129c09f3f7102be792ef360247b26d1881eaf54cd3dccd97"} Oct 31 14:54:13.261828 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:13.261792 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:13.278916 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:13.278886 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:13.900499 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:13.900469 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 14:54:13.900643 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:13.900516 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 14:54:14.070133 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:14.070088 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" event={"ID":"33d1c606-bde8-4f26-9ebb-c51a6a7ab576","Type":"ContainerStarted","Data":"f2a29c87395a9ae3b5fa6e56b5f069fed6865534cad8e614022ad7315c18782c"} Oct 31 14:54:14.081260 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:14.081239 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98" Oct 31 14:54:14.162776 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:14.160711 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:14.185606 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:14.185557 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" podStartSLOduration=2.580786592 podStartE2EDuration="3.185542708s" totalImagesPullingTime="604.756116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:14.184285169 +0000 UTC m=+497.395526581" watchObservedRunningTime="2026-10-31 14:54:14.185542708 +0000 UTC m=+497.396784172" Oct 31 14:54:14.900144 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:14.900110 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 14:54:14.900343 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:14.900172 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 14:54:15.899942 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:15.899909 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 14:54:15.900334 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:15.899957 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 14:54:16.900119 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:16.900087 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 14:54:16.900508 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:16.900130 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 14:54:17.900054 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:17.900023 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 14:54:17.900225 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:17.900080 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 14:54:18.899741 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:18.899688 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Oct 31 14:54:18.899907 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:18.899765 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Oct 31 14:54:20.900286 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:20.900248 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:54:20.900688 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:20.900321 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:54:21.941884 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:21.941836 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:54:21.942487 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:21.941895 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:54:22.142977 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:22.142944 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:22.156189 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:22.156165 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:22.778162 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:22.778128 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:22.778610 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:22.778586 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:22.779385 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:22.779365 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 14:54:23.129007 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:23.128932 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb" Oct 31 14:54:26.205646 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:54:26.205612 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:54:37.819120 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.819086 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv"] Oct 31 14:54:37.824451 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.824434 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:37.925254 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.925223 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv"] Oct 31 14:54:37.976433 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.976406 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:37.976588 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.976446 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:37.976588 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.976478 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:37.976588 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.976566 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2gms\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-kube-api-access-m2gms\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:37.976729 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.976610 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:37.976830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.976816 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:37.976891 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.976852 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:37.976891 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.976879 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2818d837-fb1b-46f8-b50b-902ac273d9e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:37.977008 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:37.976947 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078076 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.077994 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078076 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078047 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078285 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078095 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078285 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078122 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2818d837-fb1b-46f8-b50b-902ac273d9e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078285 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078156 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078285 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078213 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078285 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078244 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078285 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078278 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078571 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078309 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2gms\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-kube-api-access-m2gms\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078571 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078431 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078679 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078589 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078754 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078710 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078792 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078751 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.078970 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.078951 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2818d837-fb1b-46f8-b50b-902ac273d9e6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.080988 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.080967 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.081096 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.081081 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.135108 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.135075 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.135108 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.135098 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2gms\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-kube-api-access-m2gms\") pod \"maas-default-gateway-openshift-default-67bf986f99-qllhv\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.137013 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.136995 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:38.562684 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.562534 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv"] Oct 31 14:54:38.565384 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:54:38.565360 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2818d837_fb1b_46f8_b50b_902ac273d9e6.slice/crio-851dc19a65deccf4c66cde2028a85dbe33c0ed6e4770bf2b91096e7a3c45dcb0 WatchSource:0}: Error finding container 851dc19a65deccf4c66cde2028a85dbe33c0ed6e4770bf2b91096e7a3c45dcb0: Status 404 returned error can't find the container with id 851dc19a65deccf4c66cde2028a85dbe33c0ed6e4770bf2b91096e7a3c45dcb0 Oct 31 14:54:38.567447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.567416 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:54:38.567531 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.567493 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:54:38.567602 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:38.567539 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:54:39.190272 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:39.190240 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" event={"ID":"2818d837-fb1b-46f8-b50b-902ac273d9e6","Type":"ContainerStarted","Data":"7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08"} Oct 31 14:54:39.190677 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:39.190291 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" event={"ID":"2818d837-fb1b-46f8-b50b-902ac273d9e6","Type":"ContainerStarted","Data":"851dc19a65deccf4c66cde2028a85dbe33c0ed6e4770bf2b91096e7a3c45dcb0"} Oct 31 14:54:39.418226 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:39.418170 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podStartSLOduration=2.418156982 podStartE2EDuration="2.418156982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:39.416453226 +0000 UTC m=+522.627694638" watchObservedRunningTime="2026-10-31 14:54:39.418156982 +0000 UTC m=+522.629398394" Oct 31 14:54:40.137437 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:40.137402 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:40.138707 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:40.138667 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:40.138835 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:40.138740 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 14:54:41.137585 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:41.137550 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:41.137969 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:41.137600 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 14:54:42.137318 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:42.137284 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:42.137484 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:42.137334 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 14:54:43.137554 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:43.137527 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:43.138003 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:43.137575 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 14:54:44.137382 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:44.137342 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:44.137586 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:44.137400 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 14:54:44.879432 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:44.879397 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b9bfdbc7b-7jf7k"] Oct 31 14:54:44.885508 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:44.885484 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" Oct 31 14:54:44.940216 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:44.940179 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b9bfdbc7b-7jf7k"] Oct 31 14:54:44.949953 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:44.949913 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1404f6c8-54fc-49bc-b514-3e6956e20350-tls-cert\") pod \"authorino-b9bfdbc7b-7jf7k\" (UID: \"1404f6c8-54fc-49bc-b514-3e6956e20350\") " pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" Oct 31 14:54:44.950093 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:44.949970 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlhh\" (UniqueName: \"kubernetes.io/projected/1404f6c8-54fc-49bc-b514-3e6956e20350-kube-api-access-xqlhh\") pod \"authorino-b9bfdbc7b-7jf7k\" (UID: \"1404f6c8-54fc-49bc-b514-3e6956e20350\") " pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" Oct 31 14:54:44.990085 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:44.990053 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 14:54:44.990085 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:44.990062 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lr8w6\"" Oct 31 14:54:45.051087 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:45.051050 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1404f6c8-54fc-49bc-b514-3e6956e20350-tls-cert\") pod \"authorino-b9bfdbc7b-7jf7k\" (UID: \"1404f6c8-54fc-49bc-b514-3e6956e20350\") " pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" Oct 31 14:54:45.051246 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:45.051107 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlhh\" (UniqueName: \"kubernetes.io/projected/1404f6c8-54fc-49bc-b514-3e6956e20350-kube-api-access-xqlhh\") pod \"authorino-b9bfdbc7b-7jf7k\" (UID: \"1404f6c8-54fc-49bc-b514-3e6956e20350\") " pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" Oct 31 14:54:45.053785 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:45.053763 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1404f6c8-54fc-49bc-b514-3e6956e20350-tls-cert\") pod \"authorino-b9bfdbc7b-7jf7k\" (UID: \"1404f6c8-54fc-49bc-b514-3e6956e20350\") " pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" Oct 31 14:54:45.108248 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:45.108215 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlhh\" (UniqueName: \"kubernetes.io/projected/1404f6c8-54fc-49bc-b514-3e6956e20350-kube-api-access-xqlhh\") pod \"authorino-b9bfdbc7b-7jf7k\" (UID: \"1404f6c8-54fc-49bc-b514-3e6956e20350\") " pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" Oct 31 14:54:45.137913 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:45.137832 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:45.137913 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:45.137896 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 14:54:45.198982 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:45.198954 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" Oct 31 14:54:45.573748 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:45.573680 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b9bfdbc7b-7jf7k"] Oct 31 14:54:45.577204 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:54:45.577166 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1404f6c8_54fc_49bc_b514_3e6956e20350.slice/crio-62d777dc3222229852e438370aa8d2ca55ebc966011e0b23736d7b59a4555339 WatchSource:0}: Error finding container 62d777dc3222229852e438370aa8d2ca55ebc966011e0b23736d7b59a4555339: Status 404 returned error can't find the container with id 62d777dc3222229852e438370aa8d2ca55ebc966011e0b23736d7b59a4555339 Oct 31 14:54:46.137460 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:46.137426 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Oct 31 14:54:46.137648 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:46.137481 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Oct 31 14:54:46.226064 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:46.226028 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" event={"ID":"1404f6c8-54fc-49bc-b514-3e6956e20350","Type":"ContainerStarted","Data":"62d777dc3222229852e438370aa8d2ca55ebc966011e0b23736d7b59a4555339"} Oct 31 14:54:47.831827 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:47.831794 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:47.832298 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:47.832219 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:47.833109 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:47.833093 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 14:54:48.239227 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:48.239190 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" event={"ID":"1404f6c8-54fc-49bc-b514-3e6956e20350","Type":"ContainerStarted","Data":"c4b1b9f233352768900efc4897ba9659a70b0b5e981179c9279e5a5e26accad2"} Oct 31 14:54:48.351545 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:54:48.351489 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-b9bfdbc7b-7jf7k" podStartSLOduration=2.705931049 podStartE2EDuration="4.351467665s" totalImagesPullingTime="1.645536616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:54:48.35074366 +0000 UTC m=+531.561985072" watchObservedRunningTime="2026-10-31 14:54:48.351467665 +0000 UTC m=+531.562709080" Oct 31 14:54:56.206848 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:54:56.206805 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:55:18.671903 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.671862 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h"] Oct 31 14:55:18.675446 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.675432 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.727217 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.727192 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-33bc513e-openshift-default-dockercfg-ttl7s\"" Oct 31 14:55:18.778354 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.778327 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h"] Oct 31 14:55:18.791959 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.791920 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-socket\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.792113 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.791965 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-credential-socket\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.792113 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.792091 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-podinfo\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.792225 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.792135 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-envoy\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.792267 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.792220 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-certs\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.792318 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.792283 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9qxk\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-kube-api-access-d9qxk\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.792440 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.792424 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-data\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.792482 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.792454 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-token\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.792531 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.792488 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istiod-ca-cert\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.896414 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9qxk\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-kube-api-access-d9qxk\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.896619 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-data\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.896663 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-token\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.896726 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istiod-ca-cert\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.896770 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-socket\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.896794 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-credential-socket\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.896899 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-podinfo\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897209 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.896953 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-envoy\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897692 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.897255 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-credential-socket\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.897692 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.897437 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-certs\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.898160 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.898132 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-data\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.898284 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.898141 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-socket\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.898284 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.898197 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-certs\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.898361 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.898266 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istiod-ca-cert\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.899906 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.899879 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-envoy\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.900631 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.900606 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-podinfo\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.974592 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.974564 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-token\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.986198 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.986165 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9qxk\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-kube-api-access-d9qxk\") pod \"e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:18.989042 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:18.989026 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:19.413658 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:19.413637 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h"] Oct 31 14:55:19.415871 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:55:19.415843 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af327ec_3dc1_4682_a1cd_dc546b8afbd2.slice/crio-f21b2a6eb66cb5335c9ea191bcce01f3186692a61108fc0abb5bec0f9b34ccc2 WatchSource:0}: Error finding container f21b2a6eb66cb5335c9ea191bcce01f3186692a61108fc0abb5bec0f9b34ccc2: Status 404 returned error can't find the container with id f21b2a6eb66cb5335c9ea191bcce01f3186692a61108fc0abb5bec0f9b34ccc2 Oct 31 14:55:19.417912 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:19.417883 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:19.417990 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:19.417950 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:19.417990 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:19.417980 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:20.411004 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:20.410963 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" event={"ID":"3af327ec-3dc1-4682-a1cd-dc546b8afbd2","Type":"ContainerStarted","Data":"74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2"} Oct 31 14:55:20.411450 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:20.411031 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" event={"ID":"3af327ec-3dc1-4682-a1cd-dc546b8afbd2","Type":"ContainerStarted","Data":"f21b2a6eb66cb5335c9ea191bcce01f3186692a61108fc0abb5bec0f9b34ccc2"} Oct 31 14:55:20.577514 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:20.577459 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" podStartSLOduration=2.577442108 podStartE2EDuration="2.577442108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:20.576288526 +0000 UTC m=+563.787529940" watchObservedRunningTime="2026-10-31 14:55:20.577442108 +0000 UTC m=+563.788683521" Oct 31 14:55:20.895458 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:20.895426 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs"] Oct 31 14:55:20.899198 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:20.899182 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:20.952134 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:20.952101 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs"] Oct 31 14:55:20.989304 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:20.989273 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:20.994176 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:20.994147 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:20.999088 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:20.999067 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b671479d-openshift-default-dockercfg-s9x9g\"" Oct 31 14:55:21.016932 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.016906 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-credential-socket\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.017030 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.016953 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-podinfo\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.017070 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.017032 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-certs\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.017103 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.017075 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-data\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.017103 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.017092 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-envoy\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.017192 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.017172 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7d2d0-a49e-4039-b7b6-496017dd6607-istiod-ca-cert\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.017252 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.017237 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cztsh\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-kube-api-access-cztsh\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.017288 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.017267 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-socket\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.017359 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.017345 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-token\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.118417 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.118386 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7d2d0-a49e-4039-b7b6-496017dd6607-istiod-ca-cert\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.118417 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.118431 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cztsh\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-kube-api-access-cztsh\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.118633 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.118453 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-socket\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.118633 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.118512 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-token\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.118633 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.118560 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-credential-socket\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.118633 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.118608 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-podinfo\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.118894 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.118653 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-certs\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.118894 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.118690 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-data\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.118894 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.118858 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-envoy\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.119196 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.119168 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-socket\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.119323 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.119237 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-credential-socket\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.119323 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.119247 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-data\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.119323 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.119291 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7d2d0-a49e-4039-b7b6-496017dd6607-istiod-ca-cert\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.119485 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.119377 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-certs\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.121457 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.121436 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-envoy\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.121548 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.121506 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-podinfo\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.175686 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.175613 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-token\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.176101 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.176083 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cztsh\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-kube-api-access-cztsh\") pod \"e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.213051 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.213021 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:21.416744 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.416719 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:21.417594 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.417575 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:21.692737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.692714 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs"] Oct 31 14:55:21.694859 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:55:21.694831 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db7d2d0_a49e_4039_b7b6_496017dd6607.slice/crio-b3e5f4ae2ad5b560e0e8e7f775d1ac613868d010c034a129f6d783cef2ab11f4 WatchSource:0}: Error finding container b3e5f4ae2ad5b560e0e8e7f775d1ac613868d010c034a129f6d783cef2ab11f4: Status 404 returned error can't find the container with id b3e5f4ae2ad5b560e0e8e7f775d1ac613868d010c034a129f6d783cef2ab11f4 Oct 31 14:55:21.697008 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.696980 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:21.697087 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.697042 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:21.697087 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:21.697072 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:22.429632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:22.429593 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" event={"ID":"5db7d2d0-a49e-4039-b7b6-496017dd6607","Type":"ContainerStarted","Data":"edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa"} Oct 31 14:55:22.430080 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:22.429683 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" event={"ID":"5db7d2d0-a49e-4039-b7b6-496017dd6607","Type":"ContainerStarted","Data":"b3e5f4ae2ad5b560e0e8e7f775d1ac613868d010c034a129f6d783cef2ab11f4"} Oct 31 14:55:22.549666 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:22.549611 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" podStartSLOduration=2.549597137 podStartE2EDuration="2.549597137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:22.54847192 +0000 UTC m=+565.759713333" watchObservedRunningTime="2026-10-31 14:55:22.549597137 +0000 UTC m=+565.760838548" Oct 31 14:55:23.213777 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:23.213742 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:23.219034 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:23.219010 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:23.435041 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:23.435001 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:23.436030 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:23.436009 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:25.578169 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.578138 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j"] Oct 31 14:55:25.581770 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.581752 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.642565 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.642534 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ee109dd4-openshift-default-dockercfg-twknc\"" Oct 31 14:55:25.660011 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.659981 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-envoy\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.660162 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.660027 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istiod-ca-cert\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.660162 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.660049 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-credential-socket\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.660285 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.660202 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxxt\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-kube-api-access-2jxxt\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.660285 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.660269 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-podinfo\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.660403 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.660350 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-token\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.660403 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.660387 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-socket\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.660507 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.660441 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-data\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.660570 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.660551 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-certs\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.685384 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.685355 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j"] Oct 31 14:55:25.761492 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761460 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-certs\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.761659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761497 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-envoy\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.761659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761540 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istiod-ca-cert\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.761659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761569 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-credential-socket\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.761659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761596 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxxt\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-kube-api-access-2jxxt\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.761659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761637 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-podinfo\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.761921 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761689 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-token\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.761921 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761746 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-socket\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.761921 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761778 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-data\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.762076 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.761941 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-certs\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.762076 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.762032 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-credential-socket\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.762183 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.762133 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-data\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.762255 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.762202 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-socket\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.762461 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.762437 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istiod-ca-cert\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.763960 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.763940 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-envoy\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.764271 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.764255 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-podinfo\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.819621 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.819590 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-token\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.819921 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.819903 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxxt\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-kube-api-access-2jxxt\") pod \"e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:25.894824 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:25.894755 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:26.205899 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:55:26.205867 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:55:26.375465 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:26.375441 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j"] Oct 31 14:55:26.377872 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:55:26.377830 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185b05b1_78c0_42fe_ae36_8cb67dd406a5.slice/crio-6429db260e094a42acbd62787ec87551b5fde800f650062145de83d2dadc19fc WatchSource:0}: Error finding container 6429db260e094a42acbd62787ec87551b5fde800f650062145de83d2dadc19fc: Status 404 returned error can't find the container with id 6429db260e094a42acbd62787ec87551b5fde800f650062145de83d2dadc19fc Oct 31 14:55:26.380156 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:26.380127 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:26.380220 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:26.380192 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:26.380263 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:26.380220 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:26.450501 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:26.450473 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" event={"ID":"185b05b1-78c0-42fe-ae36-8cb67dd406a5","Type":"ContainerStarted","Data":"38b1b8705d38cf388f08e9711fb72c1edf41f99fbf8c36e7c652dc1fb4bb0403"} Oct 31 14:55:26.450616 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:26.450532 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" event={"ID":"185b05b1-78c0-42fe-ae36-8cb67dd406a5","Type":"ContainerStarted","Data":"6429db260e094a42acbd62787ec87551b5fde800f650062145de83d2dadc19fc"} Oct 31 14:55:26.566131 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:26.566029 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" podStartSLOduration=1.566013329 podStartE2EDuration="1.566013329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:26.564842273 +0000 UTC m=+569.776083709" watchObservedRunningTime="2026-10-31 14:55:26.566013329 +0000 UTC m=+569.777254740" Oct 31 14:55:26.901869 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:26.901802 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:27.900468 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:27.900437 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:28.461550 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.461516 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:28.462525 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.462508 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:55:28.573839 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.573802 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh"] Oct 31 14:55:28.577265 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.577242 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.629142 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.629119 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-33bc513e-serving-cert\"" Oct 31 14:55:28.636397 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.636368 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh"] Oct 31 14:55:28.692042 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.692013 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053490e-13c8-4dfc-b075-642565afc449-openshift-service-ca\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.692212 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.692168 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-metrics-certs\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.692271 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.692231 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjtr\" (UniqueName: \"kubernetes.io/projected/a053490e-13c8-4dfc-b075-642565afc449-kube-api-access-7bjtr\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.692320 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.692278 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.792873 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.792802 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-metrics-certs\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.792873 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.792842 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjtr\" (UniqueName: \"kubernetes.io/projected/a053490e-13c8-4dfc-b075-642565afc449-kube-api-access-7bjtr\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.793062 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.792884 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.793062 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.792910 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053490e-13c8-4dfc-b075-642565afc449-openshift-service-ca\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.793062 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:55:28.793026 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-serving-cert: secret "maas-api-e2e-ait-33bc513e-serving-cert" not found Oct 31 14:55:28.793182 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:55:28.793099 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls podName:a053490e-13c8-4dfc-b075-642565afc449 nodeName:}" failed. No retries permitted until 2026-10-31 14:55:29.293076685 +0000 UTC m=+572.504318091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls") pod "maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" (UID: "a053490e-13c8-4dfc-b075-642565afc449") : secret "maas-api-e2e-ait-33bc513e-serving-cert" not found Oct 31 14:55:28.793524 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.793506 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053490e-13c8-4dfc-b075-642565afc449-openshift-service-ca\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.795368 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.795349 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-metrics-certs\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:28.851391 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:28.851361 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjtr\" (UniqueName: \"kubernetes.io/projected/a053490e-13c8-4dfc-b075-642565afc449-kube-api-access-7bjtr\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:29.309486 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:29.309449 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:29.312023 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:29.312003 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls\") pod \"maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:29.490607 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:29.490579 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:29.864804 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:29.864737 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh"] Oct 31 14:55:29.867188 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:55:29.867157 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda053490e_13c8_4dfc_b075_642565afc449.slice/crio-f645f99e8167a01a81f8005d4ffbe2d3077e7dfa2eb6c728b79a201a087e65b7 WatchSource:0}: Error finding container f645f99e8167a01a81f8005d4ffbe2d3077e7dfa2eb6c728b79a201a087e65b7: Status 404 returned error can't find the container with id f645f99e8167a01a81f8005d4ffbe2d3077e7dfa2eb6c728b79a201a087e65b7 Oct 31 14:55:30.472056 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:30.472025 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" event={"ID":"a053490e-13c8-4dfc-b075-642565afc449","Type":"ContainerStarted","Data":"f645f99e8167a01a81f8005d4ffbe2d3077e7dfa2eb6c728b79a201a087e65b7"} Oct 31 14:55:31.482195 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:31.482150 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" event={"ID":"a053490e-13c8-4dfc-b075-642565afc449","Type":"ContainerStarted","Data":"516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06"} Oct 31 14:55:31.482719 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:31.482258 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:31.596750 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:31.596678 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" podStartSLOduration=2.637105048 podStartE2EDuration="3.596665171s" totalImagesPullingTime="959.560123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:31.595131156 +0000 UTC m=+574.806372570" watchObservedRunningTime="2026-10-31 14:55:31.596665171 +0000 UTC m=+574.807906623" Oct 31 14:55:33.806935 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.806890 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg"] Oct 31 14:55:33.811649 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.811625 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.854525 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.854498 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a78a5-39c2-4843-aaf4-31e36692861d-openshift-service-ca\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.854525 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.854526 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-metrics-certs\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.854736 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.854552 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.854736 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.854667 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmqv\" (UniqueName: \"kubernetes.io/projected/d88a78a5-39c2-4843-aaf4-31e36692861d-kube-api-access-jrmqv\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.916182 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.916146 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b671479d-serving-cert\"" Oct 31 14:55:33.918366 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.918343 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg"] Oct 31 14:55:33.972762 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.965633 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmqv\" (UniqueName: \"kubernetes.io/projected/d88a78a5-39c2-4843-aaf4-31e36692861d-kube-api-access-jrmqv\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.972762 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.965819 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a78a5-39c2-4843-aaf4-31e36692861d-openshift-service-ca\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.972762 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.965852 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-metrics-certs\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.972762 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.965878 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.972762 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:55:33.966028 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-serving-cert: secret "maas-api-e2e-ait-b671479d-serving-cert" not found Oct 31 14:55:33.972762 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:55:33.966096 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls podName:d88a78a5-39c2-4843-aaf4-31e36692861d nodeName:}" failed. No retries permitted until 2026-10-31 14:55:34.466074316 +0000 UTC m=+577.677315708 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls") pod "maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" (UID: "d88a78a5-39c2-4843-aaf4-31e36692861d") : secret "maas-api-e2e-ait-b671479d-serving-cert" not found Oct 31 14:55:33.972762 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.967190 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a78a5-39c2-4843-aaf4-31e36692861d-openshift-service-ca\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:33.975579 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:33.975505 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-metrics-certs\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:34.022618 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:34.022591 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmqv\" (UniqueName: \"kubernetes.io/projected/d88a78a5-39c2-4843-aaf4-31e36692861d-kube-api-access-jrmqv\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:34.476389 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:34.476357 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:34.479200 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:34.479164 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls\") pod \"maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:34.727855 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:34.727763 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:35.137750 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:35.132239 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg"] Oct 31 14:55:35.140029 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:55:35.140000 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88a78a5_39c2_4843_aaf4_31e36692861d.slice/crio-0b922a639dd44fc18505b0fc55b49c6697f270137ec0fab3f8460b70b64a87b7 WatchSource:0}: Error finding container 0b922a639dd44fc18505b0fc55b49c6697f270137ec0fab3f8460b70b64a87b7: Status 404 returned error can't find the container with id 0b922a639dd44fc18505b0fc55b49c6697f270137ec0fab3f8460b70b64a87b7 Oct 31 14:55:35.501551 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:35.501513 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" event={"ID":"d88a78a5-39c2-4843-aaf4-31e36692861d","Type":"ContainerStarted","Data":"0b922a639dd44fc18505b0fc55b49c6697f270137ec0fab3f8460b70b64a87b7"} Oct 31 14:55:36.508736 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:36.508677 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" event={"ID":"d88a78a5-39c2-4843-aaf4-31e36692861d","Type":"ContainerStarted","Data":"997a0605e72d538637e3efdd5f8092ae2afac8eda12951805457e57399e0a38c"} Oct 31 14:55:36.509103 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:36.508819 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:36.626457 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:36.626405 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" podStartSLOduration=2.78265611 podStartE2EDuration="3.626391216s" totalImagesPullingTime="843.735106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:36.625065037 +0000 UTC m=+579.836306448" watchObservedRunningTime="2026-10-31 14:55:36.626391216 +0000 UTC m=+579.837632627" Oct 31 14:55:37.493003 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:37.492927 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:55:42.520822 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:42.520796 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:55:43.567433 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:43.567396 4399 generic.go:356] "Generic (PLEG): container finished" podID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerID="74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2" exitCode=0 Oct 31 14:55:43.567882 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:43.567459 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" event={"ID":"3af327ec-3dc1-4682-a1cd-dc546b8afbd2","Type":"ContainerDied","Data":"74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2"} Oct 31 14:55:43.567929 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:43.567915 4399 scope.go:122] "RemoveContainer" containerID="74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2" Oct 31 14:55:43.569250 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:43.569220 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:43.569340 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:43.569289 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:43.569340 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:43.569319 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:44.594019 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:44.593977 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" event={"ID":"3af327ec-3dc1-4682-a1cd-dc546b8afbd2","Type":"ContainerStarted","Data":"cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8"} Oct 31 14:55:44.991068 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:44.990947 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:44.991409 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:44.991384 4399 patch_prober.go:29] interesting pod/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:55:44.991602 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:44.991571 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 14:55:45.990347 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:45.990312 4399 patch_prober.go:29] interesting pod/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:55:45.991022 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:45.990986 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 14:55:46.989377 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:46.989336 4399 patch_prober.go:29] interesting pod/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:55:46.989574 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:46.989393 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 14:55:47.991061 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:47.990583 4399 patch_prober.go:29] interesting pod/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:55:47.991061 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:47.990655 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 14:55:48.479312 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.479274 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg"] Oct 31 14:55:48.483568 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.483533 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.536688 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.536649 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-08b12cc9-serving-cert\"" Oct 31 14:55:48.566673 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.566322 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-openshift-service-ca\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.566673 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.566443 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-metrics-certs\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.566673 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.566510 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msl2s\" (UniqueName: \"kubernetes.io/projected/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-kube-api-access-msl2s\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.566673 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.566537 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.598760 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.598656 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg"] Oct 31 14:55:48.667393 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.667351 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-metrics-certs\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.667792 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.667768 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msl2s\" (UniqueName: \"kubernetes.io/projected/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-kube-api-access-msl2s\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.667992 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.667947 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.668245 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.668181 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-openshift-service-ca\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.668930 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:55:48.668889 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-serving-cert: secret "maas-api-e2e-ait-08b12cc9-serving-cert" not found Oct 31 14:55:48.669030 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:55:48.668998 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls podName:53c05505-a3f3-4b8c-8ad8-2bd4d114c932 nodeName:}" failed. No retries permitted until 2026-10-31 14:55:49.168959922 +0000 UTC m=+592.380201316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls") pod "maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" (UID: "53c05505-a3f3-4b8c-8ad8-2bd4d114c932") : secret "maas-api-e2e-ait-08b12cc9-serving-cert" not found Oct 31 14:55:48.669452 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.669430 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-openshift-service-ca\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.674367 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.674344 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-metrics-certs\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.729032 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.728999 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msl2s\" (UniqueName: \"kubernetes.io/projected/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-kube-api-access-msl2s\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:48.990297 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.990269 4399 patch_prober.go:29] interesting pod/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:55:48.990476 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:48.990317 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 14:55:49.174812 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:49.174772 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:49.177344 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:49.177319 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls\") pod \"maas-api-e2e-ait-08b12cc9-646487b686-f9kjg\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:49.398006 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:49.397932 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:49.798655 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:49.798626 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg"] Oct 31 14:55:49.801286 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:55:49.801257 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c05505_a3f3_4b8c_8ad8_2bd4d114c932.slice/crio-fcb606dd31ddada2df79175cc331f8e371a913e95faa1079f4d404e30388d8fd WatchSource:0}: Error finding container fcb606dd31ddada2df79175cc331f8e371a913e95faa1079f4d404e30388d8fd: Status 404 returned error can't find the container with id fcb606dd31ddada2df79175cc331f8e371a913e95faa1079f4d404e30388d8fd Oct 31 14:55:49.994017 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:49.993989 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:49.994441 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:49.994417 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:49.995185 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:49.995167 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:55:50.638088 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:50.638052 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" event={"ID":"53c05505-a3f3-4b8c-8ad8-2bd4d114c932","Type":"ContainerStarted","Data":"fcb606dd31ddada2df79175cc331f8e371a913e95faa1079f4d404e30388d8fd"} Oct 31 14:55:50.639810 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:50.639775 4399 generic.go:356] "Generic (PLEG): container finished" podID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerID="edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa" exitCode=0 Oct 31 14:55:50.640020 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:50.639845 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" event={"ID":"5db7d2d0-a49e-4039-b7b6-496017dd6607","Type":"ContainerDied","Data":"edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa"} Oct 31 14:55:50.640413 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:50.640395 4399 scope.go:122] "RemoveContainer" containerID="edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa" Oct 31 14:55:50.641608 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:50.641585 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:50.641691 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:50.641645 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:50.641691 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:50.641675 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:55:51.673031 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:51.672993 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" event={"ID":"53c05505-a3f3-4b8c-8ad8-2bd4d114c932","Type":"ContainerStarted","Data":"fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e"} Oct 31 14:55:51.673558 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:51.673198 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:55:51.675327 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:51.675295 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" event={"ID":"5db7d2d0-a49e-4039-b7b6-496017dd6607","Type":"ContainerStarted","Data":"72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d"} Oct 31 14:55:51.786801 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:51.786739 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" podStartSLOduration=2.863907739 podStartE2EDuration="3.786720136s" totalImagesPullingTime="922.812397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:51.785739454 +0000 UTC m=+594.996980863" watchObservedRunningTime="2026-10-31 14:55:51.786720136 +0000 UTC m=+594.997961545" Oct 31 14:55:52.213770 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:52.213738 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:52.214978 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:52.214947 4399 patch_prober.go:29] interesting pod/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:55:52.215106 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:52.214995 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 31 14:55:53.213949 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.213917 4399 patch_prober.go:29] interesting pod/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:55:53.214326 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.213973 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 31 14:55:53.279512 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.279477 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k"] Oct 31 14:55:53.283307 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.283285 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.324453 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.324428 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k"] Oct 31 14:55:53.371259 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.371232 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ee109dd4-serving-cert\"" Oct 31 14:55:53.424961 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.424923 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-maas-api-tls\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.425112 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.425059 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-metrics-certs\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.425168 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.425109 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d4db21-1311-49ed-a1dc-f9a38960c4ce-openshift-service-ca\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.425269 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.425250 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p89x\" (UniqueName: \"kubernetes.io/projected/55d4db21-1311-49ed-a1dc-f9a38960c4ce-kube-api-access-4p89x\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.435667 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.435653 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:53.526878 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.526784 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-metrics-certs\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.526878 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.526848 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d4db21-1311-49ed-a1dc-f9a38960c4ce-openshift-service-ca\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.527109 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.526913 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p89x\" (UniqueName: \"kubernetes.io/projected/55d4db21-1311-49ed-a1dc-f9a38960c4ce-kube-api-access-4p89x\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.527109 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.526980 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-maas-api-tls\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.527557 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.527533 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d4db21-1311-49ed-a1dc-f9a38960c4ce-openshift-service-ca\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.529537 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.529515 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-metrics-certs\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.529630 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.529574 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-maas-api-tls\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.584788 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.584756 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p89x\" (UniqueName: \"kubernetes.io/projected/55d4db21-1311-49ed-a1dc-f9a38960c4ce-kube-api-access-4p89x\") pod \"maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.596189 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.596167 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:53.994453 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:53.994431 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k"] Oct 31 14:55:53.997028 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:55:53.996993 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d4db21_1311_49ed_a1dc_f9a38960c4ce.slice/crio-59b9ab3327dcc78ebf13e2d5311a69f70ed5a8b22f71b10466c1293fd6f67f3a WatchSource:0}: Error finding container 59b9ab3327dcc78ebf13e2d5311a69f70ed5a8b22f71b10466c1293fd6f67f3a: Status 404 returned error can't find the container with id 59b9ab3327dcc78ebf13e2d5311a69f70ed5a8b22f71b10466c1293fd6f67f3a Oct 31 14:55:54.214201 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:54.214164 4399 patch_prober.go:29] interesting pod/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:55:54.214588 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:54.214219 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 31 14:55:54.700000 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:54.699967 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" event={"ID":"55d4db21-1311-49ed-a1dc-f9a38960c4ce","Type":"ContainerStarted","Data":"59b9ab3327dcc78ebf13e2d5311a69f70ed5a8b22f71b10466c1293fd6f67f3a"} Oct 31 14:55:55.213615 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:55.213584 4399 patch_prober.go:29] interesting pod/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:55:55.213855 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:55.213644 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 31 14:55:55.706990 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:55.706959 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" event={"ID":"55d4db21-1311-49ed-a1dc-f9a38960c4ce","Type":"ContainerStarted","Data":"f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4"} Oct 31 14:55:55.707475 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:55.707097 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:55:55.833055 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:55.832990 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" podStartSLOduration=2.012837521 podStartE2EDuration="2.832974751s" totalImagesPullingTime="820.13723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:55:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:55:55.830079527 +0000 UTC m=+599.041320938" watchObservedRunningTime="2026-10-31 14:55:55.832974751 +0000 UTC m=+599.044216163" Oct 31 14:55:56.206800 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:55:56.206767 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:55:56.216228 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:56.216192 4399 patch_prober.go:29] interesting pod/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Oct 31 14:55:56.216414 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:56.216272 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Oct 31 14:55:57.218273 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:57.218248 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:57.219166 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:57.219146 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:55:57.686367 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:55:57.686299 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:56:01.717989 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:01.717963 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:56:26.206014 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:56:26.205981 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:56:27.434582 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.434551 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh"] Oct 31 14:56:27.441119 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.441077 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" podUID="a053490e-13c8-4dfc-b075-642565afc449" containerName="maas-api" containerID="cri-o://516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06" gracePeriod=30 Oct 31 14:56:27.487795 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.487736 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" podUID="a053490e-13c8-4dfc-b075-642565afc449" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.45:8443/health\": dial tcp 10.133.0.45:8443: connect: connection refused" Oct 31 14:56:27.774038 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.774008 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:56:27.801262 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.801227 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls\") pod \"a053490e-13c8-4dfc-b075-642565afc449\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " Oct 31 14:56:27.801394 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.801344 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a053490e-13c8-4dfc-b075-642565afc449-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053490e-13c8-4dfc-b075-642565afc449-openshift-service-ca\") pod \"a053490e-13c8-4dfc-b075-642565afc449\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " Oct 31 14:56:27.801483 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.801467 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a053490e-13c8-4dfc-b075-642565afc449-kube-api-access-7bjtr\" (UniqueName: \"kubernetes.io/projected/a053490e-13c8-4dfc-b075-642565afc449-kube-api-access-7bjtr\") pod \"a053490e-13c8-4dfc-b075-642565afc449\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " Oct 31 14:56:27.801590 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.801561 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-metrics-certs\") pod \"a053490e-13c8-4dfc-b075-642565afc449\" (UID: \"a053490e-13c8-4dfc-b075-642565afc449\") " Oct 31 14:56:27.802212 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.802129 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a053490e-13c8-4dfc-b075-642565afc449-openshift-service-ca" pod "a053490e-13c8-4dfc-b075-642565afc449" (UID: "a053490e-13c8-4dfc-b075-642565afc449"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:27.803893 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.803870 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls" pod "a053490e-13c8-4dfc-b075-642565afc449" (UID: "a053490e-13c8-4dfc-b075-642565afc449"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:56:27.804204 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.804167 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-metrics-certs" pod "a053490e-13c8-4dfc-b075-642565afc449" (UID: "a053490e-13c8-4dfc-b075-642565afc449"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:56:27.805419 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.805393 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a053490e-13c8-4dfc-b075-642565afc449-kube-api-access-7bjtr" pod "a053490e-13c8-4dfc-b075-642565afc449" (UID: "a053490e-13c8-4dfc-b075-642565afc449"). InnerVolumeSpecName "kube-api-access-7bjtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:27.872109 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.872071 4399 generic.go:356] "Generic (PLEG): container finished" podID="a053490e-13c8-4dfc-b075-642565afc449" containerID="516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06" exitCode=0 Oct 31 14:56:27.872298 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.872122 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" Oct 31 14:56:27.872298 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.872167 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" event={"ID":"a053490e-13c8-4dfc-b075-642565afc449","Type":"ContainerDied","Data":"516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06"} Oct 31 14:56:27.872298 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.872223 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh" event={"ID":"a053490e-13c8-4dfc-b075-642565afc449","Type":"ContainerDied","Data":"f645f99e8167a01a81f8005d4ffbe2d3077e7dfa2eb6c728b79a201a087e65b7"} Oct 31 14:56:27.872298 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.872243 4399 scope.go:122] "RemoveContainer" containerID="516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06" Oct 31 14:56:27.881880 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.881858 4399 scope.go:122] "RemoveContainer" containerID="516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06" Oct 31 14:56:27.882133 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:56:27.882115 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06\": container with ID starting with 516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06 not found: ID does not exist" containerID="516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06" Oct 31 14:56:27.882191 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.882142 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06"} err="failed to get container status \"516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06\": rpc error: code = NotFound desc = could not find container \"516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06\": container with ID starting with 516b58d809cfff9e0e87db8f658b5272c04536df5be81ad509d97733a1c63c06 not found: ID does not exist" Oct 31 14:56:27.903799 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.903773 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bjtr\" (UniqueName: \"kubernetes.io/projected/a053490e-13c8-4dfc-b075-642565afc449-kube-api-access-7bjtr\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:27.903799 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.903796 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:27.903994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.903807 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a053490e-13c8-4dfc-b075-642565afc449-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:27.903994 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.903815 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a053490e-13c8-4dfc-b075-642565afc449-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:27.998448 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:27.998360 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh"] Oct 31 14:56:28.010765 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:28.007984 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-33bc513e-84965d4c79-4vzzh"] Oct 31 14:56:28.575737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:28.575688 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg"] Oct 31 14:56:28.576107 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:28.575896 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" podUID="d88a78a5-39c2-4843-aaf4-31e36692861d" containerName="maas-api" containerID="cri-o://997a0605e72d538637e3efdd5f8092ae2afac8eda12951805457e57399e0a38c" gracePeriod=30 Oct 31 14:56:28.878883 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:28.878856 4399 generic.go:356] "Generic (PLEG): container finished" podID="d88a78a5-39c2-4843-aaf4-31e36692861d" containerID="997a0605e72d538637e3efdd5f8092ae2afac8eda12951805457e57399e0a38c" exitCode=0 Oct 31 14:56:28.879013 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:28.878931 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" event={"ID":"d88a78a5-39c2-4843-aaf4-31e36692861d","Type":"ContainerDied","Data":"997a0605e72d538637e3efdd5f8092ae2afac8eda12951805457e57399e0a38c"} Oct 31 14:56:28.879082 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:28.879026 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" event={"ID":"d88a78a5-39c2-4843-aaf4-31e36692861d","Type":"ContainerDied","Data":"0b922a639dd44fc18505b0fc55b49c6697f270137ec0fab3f8460b70b64a87b7"} Oct 31 14:56:28.879082 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:28.879056 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b922a639dd44fc18505b0fc55b49c6697f270137ec0fab3f8460b70b64a87b7" Oct 31 14:56:28.893578 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:28.893557 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:56:28.894928 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:28.894910 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a053490e-13c8-4dfc-b075-642565afc449" path="/var/lib/kubelet/pods/a053490e-13c8-4dfc-b075-642565afc449/volumes" Oct 31 14:56:29.019079 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.019052 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d88a78a5-39c2-4843-aaf4-31e36692861d-kube-api-access-jrmqv\" (UniqueName: \"kubernetes.io/projected/d88a78a5-39c2-4843-aaf4-31e36692861d-kube-api-access-jrmqv\") pod \"d88a78a5-39c2-4843-aaf4-31e36692861d\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " Oct 31 14:56:29.019278 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.019118 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-metrics-certs\") pod \"d88a78a5-39c2-4843-aaf4-31e36692861d\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " Oct 31 14:56:29.019278 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.019176 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls\") pod \"d88a78a5-39c2-4843-aaf4-31e36692861d\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " Oct 31 14:56:29.019278 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.019226 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d88a78a5-39c2-4843-aaf4-31e36692861d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a78a5-39c2-4843-aaf4-31e36692861d-openshift-service-ca\") pod \"d88a78a5-39c2-4843-aaf4-31e36692861d\" (UID: \"d88a78a5-39c2-4843-aaf4-31e36692861d\") " Oct 31 14:56:29.019723 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.019660 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d88a78a5-39c2-4843-aaf4-31e36692861d-openshift-service-ca" pod "d88a78a5-39c2-4843-aaf4-31e36692861d" (UID: "d88a78a5-39c2-4843-aaf4-31e36692861d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:29.022283 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.021823 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-metrics-certs" pod "d88a78a5-39c2-4843-aaf4-31e36692861d" (UID: "d88a78a5-39c2-4843-aaf4-31e36692861d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:56:29.023927 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.023660 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d88a78a5-39c2-4843-aaf4-31e36692861d-kube-api-access-jrmqv" pod "d88a78a5-39c2-4843-aaf4-31e36692861d" (UID: "d88a78a5-39c2-4843-aaf4-31e36692861d"). InnerVolumeSpecName "kube-api-access-jrmqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:29.023927 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.023887 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls" pod "d88a78a5-39c2-4843-aaf4-31e36692861d" (UID: "d88a78a5-39c2-4843-aaf4-31e36692861d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:56:29.120808 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.120728 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrmqv\" (UniqueName: \"kubernetes.io/projected/d88a78a5-39c2-4843-aaf4-31e36692861d-kube-api-access-jrmqv\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:29.120808 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.120762 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:29.120808 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.120772 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d88a78a5-39c2-4843-aaf4-31e36692861d-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:29.120808 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.120781 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d88a78a5-39c2-4843-aaf4-31e36692861d-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:29.886269 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.886236 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg" Oct 31 14:56:29.967722 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.964670 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h"] Oct 31 14:56:29.967722 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:29.966007 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" containerID="cri-o://cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8" gracePeriod=30 Oct 31 14:56:30.082632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:30.082572 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg"] Oct 31 14:56:30.085562 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:30.085531 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b671479d-55f94fd8d8-zcmwg"] Oct 31 14:56:30.894981 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:30.894953 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d88a78a5-39c2-4843-aaf4-31e36692861d" path="/var/lib/kubelet/pods/d88a78a5-39c2-4843-aaf4-31e36692861d/volumes" Oct 31 14:56:32.422488 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:32.422459 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs"] Oct 31 14:56:32.422925 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:32.422657 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" containerID="cri-o://72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d" gracePeriod=30 Oct 31 14:56:34.995909 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:34.995881 4399 patch_prober.go:29] interesting pod/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 31 14:56:34.996351 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:34.995962 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 31 14:56:35.224827 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.224800 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:56:35.308384 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308314 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istiod-ca-cert\") pod \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " Oct 31 14:56:35.308384 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308365 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-credential-socket\") pod \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " Oct 31 14:56:35.308556 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308386 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-certs\") pod \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " Oct 31 14:56:35.308556 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308446 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-kube-api-access-d9qxk\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-kube-api-access-d9qxk\") pod \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " Oct 31 14:56:35.308556 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308543 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-token\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-token\") pod \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " Oct 31 14:56:35.308793 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308599 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-podinfo\") pod \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " Oct 31 14:56:35.308793 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308633 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-credential-socket" pod "3af327ec-3dc1-4682-a1cd-dc546b8afbd2" (UID: "3af327ec-3dc1-4682-a1cd-dc546b8afbd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:35.308793 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308643 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-socket\") pod \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " Oct 31 14:56:35.308793 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308739 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-data\") pod \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " Oct 31 14:56:35.308793 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308782 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-envoy\") pod \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\" (UID: \"3af327ec-3dc1-4682-a1cd-dc546b8afbd2\") " Oct 31 14:56:35.309040 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308867 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-socket" pod "3af327ec-3dc1-4682-a1cd-dc546b8afbd2" (UID: "3af327ec-3dc1-4682-a1cd-dc546b8afbd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:35.309040 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.308904 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istiod-ca-cert" pod "3af327ec-3dc1-4682-a1cd-dc546b8afbd2" (UID: "3af327ec-3dc1-4682-a1cd-dc546b8afbd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:35.309114 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.309038 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-data" pod "3af327ec-3dc1-4682-a1cd-dc546b8afbd2" (UID: "3af327ec-3dc1-4682-a1cd-dc546b8afbd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:35.309218 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.309198 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.309282 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.309226 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.309282 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.309241 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.309282 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.309253 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.309426 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.309281 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-certs" pod "3af327ec-3dc1-4682-a1cd-dc546b8afbd2" (UID: "3af327ec-3dc1-4682-a1cd-dc546b8afbd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:35.311384 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.311348 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-envoy" pod "3af327ec-3dc1-4682-a1cd-dc546b8afbd2" (UID: "3af327ec-3dc1-4682-a1cd-dc546b8afbd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:35.311509 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.311387 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-podinfo" pod "3af327ec-3dc1-4682-a1cd-dc546b8afbd2" (UID: "3af327ec-3dc1-4682-a1cd-dc546b8afbd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:56:35.311570 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.311504 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-kube-api-access-d9qxk" pod "3af327ec-3dc1-4682-a1cd-dc546b8afbd2" (UID: "3af327ec-3dc1-4682-a1cd-dc546b8afbd2"). InnerVolumeSpecName "kube-api-access-d9qxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:35.311650 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.311625 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-token" pod "3af327ec-3dc1-4682-a1cd-dc546b8afbd2" (UID: "3af327ec-3dc1-4682-a1cd-dc546b8afbd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:35.409867 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.409838 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.409867 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.409864 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.409867 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.409873 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.410086 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.409882 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9qxk\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-kube-api-access-d9qxk\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.410086 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.409892 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3af327ec-3dc1-4682-a1cd-dc546b8afbd2-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:35.919908 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.919874 4399 generic.go:356] "Generic (PLEG): container finished" podID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerID="cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8" exitCode=0 Oct 31 14:56:35.920100 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.919948 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" Oct 31 14:56:35.920100 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.919975 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" event={"ID":"3af327ec-3dc1-4682-a1cd-dc546b8afbd2","Type":"ContainerDied","Data":"cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8"} Oct 31 14:56:35.920100 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.920043 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h" event={"ID":"3af327ec-3dc1-4682-a1cd-dc546b8afbd2","Type":"ContainerDied","Data":"f21b2a6eb66cb5335c9ea191bcce01f3186692a61108fc0abb5bec0f9b34ccc2"} Oct 31 14:56:35.920100 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.920071 4399 scope.go:122] "RemoveContainer" containerID="cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8" Oct 31 14:56:35.933679 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.933394 4399 scope.go:122] "RemoveContainer" containerID="74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2" Oct 31 14:56:35.942680 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.942665 4399 scope.go:122] "RemoveContainer" containerID="cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8" Oct 31 14:56:35.943018 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:56:35.942995 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8\": container with ID starting with cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8 not found: ID does not exist" containerID="cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8" Oct 31 14:56:35.943070 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.943027 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8"} err="failed to get container status \"cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8\": rpc error: code = NotFound desc = could not find container \"cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8\": container with ID starting with cbef77ee5fba0ec081f5d49780a2c78bd5ee242054ab79898311a4bc1eef2cd8 not found: ID does not exist" Oct 31 14:56:35.943070 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.943050 4399 scope.go:122] "RemoveContainer" containerID="74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2" Oct 31 14:56:35.943307 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:56:35.943287 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2\": container with ID starting with 74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2 not found: ID does not exist" containerID="74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2" Oct 31 14:56:35.943368 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:35.943317 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2"} err="failed to get container status \"74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2\": rpc error: code = NotFound desc = could not find container \"74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2\": container with ID starting with 74768886e534d29665ce43b40e57a9a9cf9fc5c532143e83ef21fc8356d5efb2 not found: ID does not exist" Oct 31 14:56:36.133759 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:36.133735 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h"] Oct 31 14:56:36.138634 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:36.138613 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-33bc513e-openshift-default-68fb8c5b9f-kmn7h"] Oct 31 14:56:36.895282 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:36.895252 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" path="/var/lib/kubelet/pods/3af327ec-3dc1-4682-a1cd-dc546b8afbd2/volumes" Oct 31 14:56:36.925903 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:36.925876 4399 generic.go:356] "Generic (PLEG): container finished" podID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerID="38b1b8705d38cf388f08e9711fb72c1edf41f99fbf8c36e7c652dc1fb4bb0403" exitCode=0 Oct 31 14:56:36.926063 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:36.925952 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" event={"ID":"185b05b1-78c0-42fe-ae36-8cb67dd406a5","Type":"ContainerDied","Data":"38b1b8705d38cf388f08e9711fb72c1edf41f99fbf8c36e7c652dc1fb4bb0403"} Oct 31 14:56:36.926318 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:36.926300 4399 scope.go:122] "RemoveContainer" containerID="38b1b8705d38cf388f08e9711fb72c1edf41f99fbf8c36e7c652dc1fb4bb0403" Oct 31 14:56:36.927458 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:36.927436 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:56:36.927517 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:36.927505 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:56:36.927564 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:36.927552 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:56:37.708013 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.707990 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:56:37.835977 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.835948 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-kube-api-access-cztsh\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-kube-api-access-cztsh\") pod \"5db7d2d0-a49e-4039-b7b6-496017dd6607\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " Oct 31 14:56:37.836165 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836076 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-socket\") pod \"5db7d2d0-a49e-4039-b7b6-496017dd6607\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " Oct 31 14:56:37.836165 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836109 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-certs\") pod \"5db7d2d0-a49e-4039-b7b6-496017dd6607\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " Oct 31 14:56:37.836165 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836155 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-envoy\") pod \"5db7d2d0-a49e-4039-b7b6-496017dd6607\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " Oct 31 14:56:37.836343 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836212 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-podinfo\") pod \"5db7d2d0-a49e-4039-b7b6-496017dd6607\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " Oct 31 14:56:37.836343 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836258 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db7d2d0-a49e-4039-b7b6-496017dd6607-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7d2d0-a49e-4039-b7b6-496017dd6607-istiod-ca-cert\") pod \"5db7d2d0-a49e-4039-b7b6-496017dd6607\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " Oct 31 14:56:37.836343 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836289 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-token\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-token\") pod \"5db7d2d0-a49e-4039-b7b6-496017dd6607\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " Oct 31 14:56:37.836483 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836340 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-data\") pod \"5db7d2d0-a49e-4039-b7b6-496017dd6607\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " Oct 31 14:56:37.836483 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836383 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-certs" pod "5db7d2d0-a49e-4039-b7b6-496017dd6607" (UID: "5db7d2d0-a49e-4039-b7b6-496017dd6607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:37.836483 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836395 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-credential-socket\") pod \"5db7d2d0-a49e-4039-b7b6-496017dd6607\" (UID: \"5db7d2d0-a49e-4039-b7b6-496017dd6607\") " Oct 31 14:56:37.836483 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836421 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-socket" pod "5db7d2d0-a49e-4039-b7b6-496017dd6607" (UID: "5db7d2d0-a49e-4039-b7b6-496017dd6607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:37.836768 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836743 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db7d2d0-a49e-4039-b7b6-496017dd6607-istiod-ca-cert" pod "5db7d2d0-a49e-4039-b7b6-496017dd6607" (UID: "5db7d2d0-a49e-4039-b7b6-496017dd6607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:56:37.837021 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836789 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:37.837140 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.837037 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:37.837140 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836906 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-data" pod "5db7d2d0-a49e-4039-b7b6-496017dd6607" (UID: "5db7d2d0-a49e-4039-b7b6-496017dd6607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:37.837140 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.836961 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-credential-socket" pod "5db7d2d0-a49e-4039-b7b6-496017dd6607" (UID: "5db7d2d0-a49e-4039-b7b6-496017dd6607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:37.839181 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.839153 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-podinfo" pod "5db7d2d0-a49e-4039-b7b6-496017dd6607" (UID: "5db7d2d0-a49e-4039-b7b6-496017dd6607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:56:37.839286 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.839243 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-envoy" pod "5db7d2d0-a49e-4039-b7b6-496017dd6607" (UID: "5db7d2d0-a49e-4039-b7b6-496017dd6607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:56:37.839409 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.839385 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-kube-api-access-cztsh" pod "5db7d2d0-a49e-4039-b7b6-496017dd6607" (UID: "5db7d2d0-a49e-4039-b7b6-496017dd6607"). InnerVolumeSpecName "kube-api-access-cztsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:37.867008 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.866940 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-token" pod "5db7d2d0-a49e-4039-b7b6-496017dd6607" (UID: "5db7d2d0-a49e-4039-b7b6-496017dd6607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:56:37.934562 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.934521 4399 generic.go:356] "Generic (PLEG): container finished" podID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerID="72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d" exitCode=0 Oct 31 14:56:37.934766 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.934581 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" Oct 31 14:56:37.934766 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.934609 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" event={"ID":"5db7d2d0-a49e-4039-b7b6-496017dd6607","Type":"ContainerDied","Data":"72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d"} Oct 31 14:56:37.934766 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.934666 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs" event={"ID":"5db7d2d0-a49e-4039-b7b6-496017dd6607","Type":"ContainerDied","Data":"b3e5f4ae2ad5b560e0e8e7f775d1ac613868d010c034a129f6d783cef2ab11f4"} Oct 31 14:56:37.934766 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.934725 4399 scope.go:122] "RemoveContainer" containerID="72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d" Oct 31 14:56:37.936830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.936807 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" event={"ID":"185b05b1-78c0-42fe-ae36-8cb67dd406a5","Type":"ContainerStarted","Data":"523bc1c82eee3372d259c0ba698f3d0cc1f217015ccd00f67b6f6dbe85713ccf"} Oct 31 14:56:37.937849 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.937830 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:37.937969 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.937853 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:37.937969 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.937867 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db7d2d0-a49e-4039-b7b6-496017dd6607-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:37.937969 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.937905 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:37.937969 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.937920 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:37.937969 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.937935 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db7d2d0-a49e-4039-b7b6-496017dd6607-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:37.937969 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.937948 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cztsh\" (UniqueName: \"kubernetes.io/projected/5db7d2d0-a49e-4039-b7b6-496017dd6607-kube-api-access-cztsh\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:56:37.972653 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.972628 4399 scope.go:122] "RemoveContainer" containerID="edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa" Oct 31 14:56:37.983182 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.983158 4399 scope.go:122] "RemoveContainer" containerID="72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d" Oct 31 14:56:37.983506 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:56:37.983482 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d\": container with ID starting with 72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d not found: ID does not exist" containerID="72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d" Oct 31 14:56:37.983590 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.983520 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d"} err="failed to get container status \"72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d\": rpc error: code = NotFound desc = could not find container \"72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d\": container with ID starting with 72876c19788d1fba641ccc9024b3c40882709f8ad6346643d7b6517eb90e051d not found: ID does not exist" Oct 31 14:56:37.983590 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.983545 4399 scope.go:122] "RemoveContainer" containerID="edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa" Oct 31 14:56:37.983855 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:56:37.983836 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa\": container with ID starting with edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa not found: ID does not exist" containerID="edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa" Oct 31 14:56:37.983906 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:37.983862 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa"} err="failed to get container status \"edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa\": rpc error: code = NotFound desc = could not find container \"edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa\": container with ID starting with edfd9da3e8b9f83ed7826020adf2970d054ccfe0cea16a713266ffc831894faa not found: ID does not exist" Oct 31 14:56:38.098262 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.098227 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs"] Oct 31 14:56:38.104770 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.104742 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b671479d-openshift-default-579c5c5897-jf6xs"] Oct 31 14:56:38.666266 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.666234 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f"] Oct 31 14:56:38.667422 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667391 4399 cpu_manager.go:414] "containerMap: removing container" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" Oct 31 14:56:38.667422 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667419 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" Oct 31 14:56:38.667632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667436 4399 cpu_manager.go:414] "containerMap: removing container" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" Oct 31 14:56:38.667632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667445 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" Oct 31 14:56:38.667632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667488 4399 cpu_manager.go:414] "containerMap: removing container" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" Oct 31 14:56:38.667632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667496 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" Oct 31 14:56:38.667632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667526 4399 cpu_manager.go:414] "containerMap: removing container" podUID="a053490e-13c8-4dfc-b075-642565afc449" containerName="maas-api" Oct 31 14:56:38.667632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667533 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a053490e-13c8-4dfc-b075-642565afc449" containerName="maas-api" Oct 31 14:56:38.667632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667545 4399 cpu_manager.go:414] "containerMap: removing container" podUID="d88a78a5-39c2-4843-aaf4-31e36692861d" containerName="maas-api" Oct 31 14:56:38.667632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667552 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d88a78a5-39c2-4843-aaf4-31e36692861d" containerName="maas-api" Oct 31 14:56:38.667998 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667681 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" Oct 31 14:56:38.667998 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667717 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a053490e-13c8-4dfc-b075-642565afc449" containerName="maas-api" Oct 31 14:56:38.667998 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667727 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d88a78a5-39c2-4843-aaf4-31e36692861d" containerName="maas-api" Oct 31 14:56:38.667998 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667738 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" Oct 31 14:56:38.667998 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667747 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" containerName="istio-proxy" Oct 31 14:56:38.667998 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.667756 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" Oct 31 14:56:38.672688 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.672669 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.728365 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.728335 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-190693-openshift-default-dockercfg-qdhg8\"" Oct 31 14:56:38.728851 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.728833 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f"] Oct 31 14:56:38.769849 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.769819 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.770024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.769859 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.770024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.769877 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.770024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.769970 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.770139 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.770093 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.770240 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.770220 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.770281 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.770262 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.770314 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.770299 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfwl7\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-kube-api-access-nfwl7\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.770437 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.770418 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.871471 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871440 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.871471 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871475 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.871737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871499 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfwl7\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-kube-api-access-nfwl7\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.871737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871551 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.871737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871573 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.871737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871604 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.871737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871629 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.871737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871653 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.871737 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871689 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.872159 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.871965 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.872159 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.872038 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.872159 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.872079 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.872159 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.872118 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.872369 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.872260 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.874128 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.874102 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.874309 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.874294 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.895321 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.895298 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db7d2d0-a49e-4039-b7b6-496017dd6607" path="/var/lib/kubelet/pods/5db7d2d0-a49e-4039-b7b6-496017dd6607/volumes" Oct 31 14:56:38.895581 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.895568 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:56:38.896182 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.896162 4399 patch_prober.go:29] interesting pod/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:38.896260 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.896215 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 31 14:56:38.929751 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.929684 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.929936 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.929916 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfwl7\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-kube-api-access-nfwl7\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-ht48f\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:38.985865 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:38.985825 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:39.428448 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.428409 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f"] Oct 31 14:56:39.431472 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:56:39.431441 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6207a0ed_36ad_4ef4_aa6a_074f0bc1b300.slice/crio-a52fe226b93f28cc26e361f2740ca6ebfc7ee7b34c4af1f44f914013f28aa088 WatchSource:0}: Error finding container a52fe226b93f28cc26e361f2740ca6ebfc7ee7b34c4af1f44f914013f28aa088: Status 404 returned error can't find the container with id a52fe226b93f28cc26e361f2740ca6ebfc7ee7b34c4af1f44f914013f28aa088 Oct 31 14:56:39.433571 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.433542 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:56:39.433645 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.433605 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:56:39.433645 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.433634 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:56:39.873728 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.870237 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5"] Oct 31 14:56:39.885554 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.883052 4399 cpu_manager.go:414] "containerMap: removing container" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" Oct 31 14:56:39.885554 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.883079 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af327ec-3dc1-4682-a1cd-dc546b8afbd2" containerName="istio-proxy" Oct 31 14:56:39.886961 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.886904 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:39.895677 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.895657 4399 patch_prober.go:29] interesting pod/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:39.895843 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.895739 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 31 14:56:39.928162 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.928132 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5"] Oct 31 14:56:39.938735 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.938650 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-029d26\"" Oct 31 14:56:39.939569 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.939552 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-029d26-dockercfg-2tgzt\"" Oct 31 14:56:39.979138 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.979101 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" event={"ID":"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300","Type":"ContainerStarted","Data":"f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0"} Oct 31 14:56:39.979335 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.979153 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" event={"ID":"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300","Type":"ContainerStarted","Data":"a52fe226b93f28cc26e361f2740ca6ebfc7ee7b34c4af1f44f914013f28aa088"} Oct 31 14:56:39.985315 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.985280 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486ec0f-d3a7-4625-8bae-1dca830110d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5\" (UID: \"c486ec0f-d3a7-4625-8bae-1dca830110d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:39.986141 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:39.986120 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:40.086763 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.086732 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6k2q\" (UniqueName: \"kubernetes.io/projected/c486ec0f-d3a7-4625-8bae-1dca830110d2-kube-api-access-p6k2q\") pod \"payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5\" (UID: \"c486ec0f-d3a7-4625-8bae-1dca830110d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:40.086945 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.086925 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486ec0f-d3a7-4625-8bae-1dca830110d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5\" (UID: \"c486ec0f-d3a7-4625-8bae-1dca830110d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:40.087558 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.087534 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486ec0f-d3a7-4625-8bae-1dca830110d2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5\" (UID: \"c486ec0f-d3a7-4625-8bae-1dca830110d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:40.129779 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.129656 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" podStartSLOduration=2.129639252 podStartE2EDuration="2.129639252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:40.12717984 +0000 UTC m=+643.338421252" watchObservedRunningTime="2026-10-31 14:56:40.129639252 +0000 UTC m=+643.340880666" Oct 31 14:56:40.188019 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.187977 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6k2q\" (UniqueName: \"kubernetes.io/projected/c486ec0f-d3a7-4625-8bae-1dca830110d2-kube-api-access-p6k2q\") pod \"payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5\" (UID: \"c486ec0f-d3a7-4625-8bae-1dca830110d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:40.262724 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.250405 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6k2q\" (UniqueName: \"kubernetes.io/projected/c486ec0f-d3a7-4625-8bae-1dca830110d2-kube-api-access-p6k2q\") pod \"payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5\" (UID: \"c486ec0f-d3a7-4625-8bae-1dca830110d2\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:40.503030 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.502997 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:40.895221 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.895125 4399 patch_prober.go:29] interesting pod/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:40.895221 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.895177 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 31 14:56:40.901845 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:56:40.901818 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc486ec0f_d3a7_4625_8bae_1dca830110d2.slice/crio-8f9bc7e62218953e193ac1d6561ec991859c8bd69d2afdacb03e576438835f19 WatchSource:0}: Error finding container 8f9bc7e62218953e193ac1d6561ec991859c8bd69d2afdacb03e576438835f19: Status 404 returned error can't find the container with id 8f9bc7e62218953e193ac1d6561ec991859c8bd69d2afdacb03e576438835f19 Oct 31 14:56:40.903614 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.903586 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5"] Oct 31 14:56:40.986099 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.986071 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" event={"ID":"c486ec0f-d3a7-4625-8bae-1dca830110d2","Type":"ContainerStarted","Data":"8f9bc7e62218953e193ac1d6561ec991859c8bd69d2afdacb03e576438835f19"} Oct 31 14:56:40.991781 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:40.991759 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:41.895237 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:41.895203 4399 patch_prober.go:29] interesting pod/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:41.895592 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:41.895254 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 31 14:56:41.992717 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:41.992669 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" event={"ID":"c486ec0f-d3a7-4625-8bae-1dca830110d2","Type":"ContainerStarted","Data":"1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8"} Oct 31 14:56:41.992912 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:41.992760 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:41.993055 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:41.993031 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:41.993811 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:41.993796 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:56:42.120531 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:42.120473 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" podStartSLOduration=3.120455189 podStartE2EDuration="3.120455189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:42.119082623 +0000 UTC m=+645.330324036" watchObservedRunningTime="2026-10-31 14:56:42.120455189 +0000 UTC m=+645.331696604" Oct 31 14:56:42.895447 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:42.895415 4399 patch_prober.go:29] interesting pod/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 31 14:56:42.895870 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:42.895478 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 31 14:56:43.462425 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:43.462395 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:56:43.899721 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:43.899623 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:56:44.005664 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:44.005639 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:56:53.001883 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:53.001856 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 14:56:56.205532 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:56:56.205495 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:56:56.760294 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.760262 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd"] Oct 31 14:56:56.773779 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.773758 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.820933 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.820909 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd"] Oct 31 14:56:56.854858 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.854836 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-metrics-certs\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.854969 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.854864 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88dns\" (UniqueName: \"kubernetes.io/projected/60c8879b-fee1-48f8-8b51-3901a4dea925-kube-api-access-88dns\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.855044 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.855019 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.855149 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.855132 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c8879b-fee1-48f8-8b51-3901a4dea925-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.867407 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.867388 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-56bc96-serving-cert\"" Oct 31 14:56:56.955715 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.955670 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c8879b-fee1-48f8-8b51-3901a4dea925-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.955896 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.955748 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-metrics-certs\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.955896 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.955775 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88dns\" (UniqueName: \"kubernetes.io/projected/60c8879b-fee1-48f8-8b51-3901a4dea925-kube-api-access-88dns\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.955896 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.955874 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.956070 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:56:56.955975 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-serving-cert: secret "maas-api-e2e-worker-w0-56bc96-serving-cert" not found Oct 31 14:56:56.956070 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:56:56.956042 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls podName:60c8879b-fee1-48f8-8b51-3901a4dea925 nodeName:}" failed. No retries permitted until 2026-10-31 14:56:57.456022008 +0000 UTC m=+660.667263414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls") pod "maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" (UID: "60c8879b-fee1-48f8-8b51-3901a4dea925") : secret "maas-api-e2e-worker-w0-56bc96-serving-cert" not found Oct 31 14:56:56.956435 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.956408 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c8879b-fee1-48f8-8b51-3901a4dea925-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:56.958623 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:56.958597 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-metrics-certs\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:57.014424 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:57.014353 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88dns\" (UniqueName: \"kubernetes.io/projected/60c8879b-fee1-48f8-8b51-3901a4dea925-kube-api-access-88dns\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:57.460631 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:57.460605 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:57.463080 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:57.463060 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls\") pod \"maas-api-e2e-worker-w0-56bc96-568b98b895-299hd\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:57.686448 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:57.686420 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:58.062866 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:58.062836 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd"] Oct 31 14:56:58.067009 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:56:58.066979 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c8879b_fee1_48f8_8b51_3901a4dea925.slice/crio-bdb2300cb07187fc18425fb18dc8c1c086e0451fc816675e792d0dcb9c18d27a WatchSource:0}: Error finding container bdb2300cb07187fc18425fb18dc8c1c086e0451fc816675e792d0dcb9c18d27a: Status 404 returned error can't find the container with id bdb2300cb07187fc18425fb18dc8c1c086e0451fc816675e792d0dcb9c18d27a Oct 31 14:56:58.068264 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:58.068244 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:56:58.074960 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:58.074937 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" event={"ID":"60c8879b-fee1-48f8-8b51-3901a4dea925","Type":"ContainerStarted","Data":"bdb2300cb07187fc18425fb18dc8c1c086e0451fc816675e792d0dcb9c18d27a"} Oct 31 14:56:59.082296 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:59.082260 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" event={"ID":"60c8879b-fee1-48f8-8b51-3901a4dea925","Type":"ContainerStarted","Data":"a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864"} Oct 31 14:56:59.082685 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:59.082327 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:56:59.199305 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:56:59.199255 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" podStartSLOduration=2.37061769 podStartE2EDuration="3.199238857s" totalImagesPullingTime="828.621167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:56:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:56:59.198421318 +0000 UTC m=+662.409662737" watchObservedRunningTime="2026-10-31 14:56:59.199238857 +0000 UTC m=+662.410480268" Oct 31 14:57:01.181771 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.181736 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg"] Oct 31 14:57:01.182263 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.181986 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" podUID="53c05505-a3f3-4b8c-8ad8-2bd4d114c932" containerName="maas-api" containerID="cri-o://fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e" gracePeriod=30 Oct 31 14:57:01.454954 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.454932 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:57:01.604180 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.604139 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-kube-api-access-msl2s\" (UniqueName: \"kubernetes.io/projected/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-kube-api-access-msl2s\") pod \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " Oct 31 14:57:01.604357 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.604279 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-openshift-service-ca\") pod \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " Oct 31 14:57:01.604357 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.604351 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-metrics-certs\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-metrics-certs\") pod \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " Oct 31 14:57:01.604499 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.604428 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls\") pod \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\" (UID: \"53c05505-a3f3-4b8c-8ad8-2bd4d114c932\") " Oct 31 14:57:01.604859 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.604673 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-openshift-service-ca" pod "53c05505-a3f3-4b8c-8ad8-2bd4d114c932" (UID: "53c05505-a3f3-4b8c-8ad8-2bd4d114c932"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:57:01.604859 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.604831 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:01.606724 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.606663 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-metrics-certs" pod "53c05505-a3f3-4b8c-8ad8-2bd4d114c932" (UID: "53c05505-a3f3-4b8c-8ad8-2bd4d114c932"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:57:01.606939 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.606908 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls" pod "53c05505-a3f3-4b8c-8ad8-2bd4d114c932" (UID: "53c05505-a3f3-4b8c-8ad8-2bd4d114c932"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:57:01.608129 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.608101 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-kube-api-access-msl2s" pod "53c05505-a3f3-4b8c-8ad8-2bd4d114c932" (UID: "53c05505-a3f3-4b8c-8ad8-2bd4d114c932"). InnerVolumeSpecName "kube-api-access-msl2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:57:01.705787 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.705759 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msl2s\" (UniqueName: \"kubernetes.io/projected/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-kube-api-access-msl2s\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:01.705787 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.705784 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:01.705974 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:01.705793 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/53c05505-a3f3-4b8c-8ad8-2bd4d114c932-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:02.099029 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.098934 4399 generic.go:356] "Generic (PLEG): container finished" podID="53c05505-a3f3-4b8c-8ad8-2bd4d114c932" containerID="fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e" exitCode=0 Oct 31 14:57:02.099174 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.099125 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" event={"ID":"53c05505-a3f3-4b8c-8ad8-2bd4d114c932","Type":"ContainerDied","Data":"fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e"} Oct 31 14:57:02.099233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.099178 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" event={"ID":"53c05505-a3f3-4b8c-8ad8-2bd4d114c932","Type":"ContainerDied","Data":"fcb606dd31ddada2df79175cc331f8e371a913e95faa1079f4d404e30388d8fd"} Oct 31 14:57:02.099233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.099212 4399 scope.go:122] "RemoveContainer" containerID="fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e" Oct 31 14:57:02.099377 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.099359 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg" Oct 31 14:57:02.119347 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.119327 4399 scope.go:122] "RemoveContainer" containerID="fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e" Oct 31 14:57:02.119745 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:57:02.119721 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e\": container with ID starting with fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e not found: ID does not exist" containerID="fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e" Oct 31 14:57:02.119822 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.119757 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e"} err="failed to get container status \"fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e\": rpc error: code = NotFound desc = could not find container \"fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e\": container with ID starting with fc394847773024df17a5d598b2018210b6b4f192bbe24bff4d470fb01a0c310e not found: ID does not exist" Oct 31 14:57:02.270563 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.270531 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg"] Oct 31 14:57:02.320094 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.320066 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-08b12cc9-646487b686-f9kjg"] Oct 31 14:57:02.791050 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.791021 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g"] Oct 31 14:57:02.791480 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.791468 4399 cpu_manager.go:414] "containerMap: removing container" podUID="53c05505-a3f3-4b8c-8ad8-2bd4d114c932" containerName="maas-api" Oct 31 14:57:02.791525 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.791481 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53c05505-a3f3-4b8c-8ad8-2bd4d114c932" containerName="maas-api" Oct 31 14:57:02.791582 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.791563 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="53c05505-a3f3-4b8c-8ad8-2bd4d114c932" containerName="maas-api" Oct 31 14:57:02.795816 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.795802 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:02.851397 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.851371 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-190693-serving-cert\"" Oct 31 14:57:02.853007 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.852984 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g"] Oct 31 14:57:02.896327 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.896293 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53c05505-a3f3-4b8c-8ad8-2bd4d114c932" path="/var/lib/kubelet/pods/53c05505-a3f3-4b8c-8ad8-2bd4d114c932/volumes" Oct 31 14:57:02.916419 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.916399 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:02.916544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.916435 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-metrics-certs\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:02.916544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.916458 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99m6t\" (UniqueName: \"kubernetes.io/projected/18f3e076-07a1-48a7-9102-273de8d907e4-kube-api-access-99m6t\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:02.916544 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:02.916476 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f3e076-07a1-48a7-9102-273de8d907e4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.017217 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.017189 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.017360 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.017235 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-metrics-certs\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.017360 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.017269 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99m6t\" (UniqueName: \"kubernetes.io/projected/18f3e076-07a1-48a7-9102-273de8d907e4-kube-api-access-99m6t\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.017500 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:57:03.017390 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-serving-cert: secret "maas-api-e2e-worker-w3-190693-serving-cert" not found Oct 31 14:57:03.017500 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:57:03.017467 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls podName:18f3e076-07a1-48a7-9102-273de8d907e4 nodeName:}" failed. No retries permitted until 2026-10-31 14:57:03.517446118 +0000 UTC m=+666.728687518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls") pod "maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" (UID: "18f3e076-07a1-48a7-9102-273de8d907e4") : secret "maas-api-e2e-worker-w3-190693-serving-cert" not found Oct 31 14:57:03.017612 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.017553 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f3e076-07a1-48a7-9102-273de8d907e4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.018339 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.018311 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f3e076-07a1-48a7-9102-273de8d907e4-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.020082 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.020063 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-metrics-certs\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.074573 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.074504 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99m6t\" (UniqueName: \"kubernetes.io/projected/18f3e076-07a1-48a7-9102-273de8d907e4-kube-api-access-99m6t\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.523481 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.523443 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.526037 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.526015 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls\") pod \"maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:03.709079 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:03.709042 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:04.083347 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:04.083325 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g"] Oct 31 14:57:04.085981 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:57:04.085954 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f3e076_07a1_48a7_9102_273de8d907e4.slice/crio-86dd2fa604673969aa4a61e7680cd02e92becb660d8631b799d72f6a897d84c8 WatchSource:0}: Error finding container 86dd2fa604673969aa4a61e7680cd02e92becb660d8631b799d72f6a897d84c8: Status 404 returned error can't find the container with id 86dd2fa604673969aa4a61e7680cd02e92becb660d8631b799d72f6a897d84c8 Oct 31 14:57:04.110314 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:04.110285 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" event={"ID":"18f3e076-07a1-48a7-9102-273de8d907e4","Type":"ContainerStarted","Data":"86dd2fa604673969aa4a61e7680cd02e92becb660d8631b799d72f6a897d84c8"} Oct 31 14:57:05.092993 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:05.092962 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 14:57:05.116938 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:05.116910 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" event={"ID":"18f3e076-07a1-48a7-9102-273de8d907e4","Type":"ContainerStarted","Data":"d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8"} Oct 31 14:57:05.117108 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:05.116977 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:05.319763 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:05.319556 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" podStartSLOduration=2.485197175 podStartE2EDuration="3.319537006s" totalImagesPullingTime="834.339831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:05.318752579 +0000 UTC m=+668.529993996" watchObservedRunningTime="2026-10-31 14:57:05.319537006 +0000 UTC m=+668.530778420" Oct 31 14:57:07.577633 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.577554 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k"] Oct 31 14:57:07.578185 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.577778 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" podUID="55d4db21-1311-49ed-a1dc-f9a38960c4ce" containerName="maas-api" containerID="cri-o://f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4" gracePeriod=30 Oct 31 14:57:07.826620 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.826579 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:57:07.968562 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.968532 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-maas-api-tls\") pod \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " Oct 31 14:57:07.968562 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.968564 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-metrics-certs\") pod \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " Oct 31 14:57:07.968844 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.968596 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d4db21-1311-49ed-a1dc-f9a38960c4ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d4db21-1311-49ed-a1dc-f9a38960c4ce-openshift-service-ca\") pod \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " Oct 31 14:57:07.968844 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.968640 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d4db21-1311-49ed-a1dc-f9a38960c4ce-kube-api-access-4p89x\" (UniqueName: \"kubernetes.io/projected/55d4db21-1311-49ed-a1dc-f9a38960c4ce-kube-api-access-4p89x\") pod \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\" (UID: \"55d4db21-1311-49ed-a1dc-f9a38960c4ce\") " Oct 31 14:57:07.969257 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.969226 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d4db21-1311-49ed-a1dc-f9a38960c4ce-openshift-service-ca" pod "55d4db21-1311-49ed-a1dc-f9a38960c4ce" (UID: "55d4db21-1311-49ed-a1dc-f9a38960c4ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:57:07.970949 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.970920 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-maas-api-tls" pod "55d4db21-1311-49ed-a1dc-f9a38960c4ce" (UID: "55d4db21-1311-49ed-a1dc-f9a38960c4ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:57:07.971051 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.971004 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d4db21-1311-49ed-a1dc-f9a38960c4ce-kube-api-access-4p89x" pod "55d4db21-1311-49ed-a1dc-f9a38960c4ce" (UID: "55d4db21-1311-49ed-a1dc-f9a38960c4ce"). InnerVolumeSpecName "kube-api-access-4p89x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:57:07.971105 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:07.971048 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-metrics-certs" pod "55d4db21-1311-49ed-a1dc-f9a38960c4ce" (UID: "55d4db21-1311-49ed-a1dc-f9a38960c4ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:57:08.070619 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.070588 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:08.070619 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.070613 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55d4db21-1311-49ed-a1dc-f9a38960c4ce-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:08.070619 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.070623 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d4db21-1311-49ed-a1dc-f9a38960c4ce-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:08.070858 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.070634 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p89x\" (UniqueName: \"kubernetes.io/projected/55d4db21-1311-49ed-a1dc-f9a38960c4ce-kube-api-access-4p89x\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:08.138868 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.138832 4399 generic.go:356] "Generic (PLEG): container finished" podID="55d4db21-1311-49ed-a1dc-f9a38960c4ce" containerID="f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4" exitCode=0 Oct 31 14:57:08.139044 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.138893 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" Oct 31 14:57:08.139044 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.138920 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" event={"ID":"55d4db21-1311-49ed-a1dc-f9a38960c4ce","Type":"ContainerDied","Data":"f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4"} Oct 31 14:57:08.139044 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.139003 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k" event={"ID":"55d4db21-1311-49ed-a1dc-f9a38960c4ce","Type":"ContainerDied","Data":"59b9ab3327dcc78ebf13e2d5311a69f70ed5a8b22f71b10466c1293fd6f67f3a"} Oct 31 14:57:08.139218 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.139069 4399 scope.go:122] "RemoveContainer" containerID="f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4" Oct 31 14:57:08.164777 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.164754 4399 scope.go:122] "RemoveContainer" containerID="f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4" Oct 31 14:57:08.165179 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:57:08.165157 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4\": container with ID starting with f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4 not found: ID does not exist" containerID="f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4" Oct 31 14:57:08.165267 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.165186 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4"} err="failed to get container status \"f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4\": rpc error: code = NotFound desc = could not find container \"f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4\": container with ID starting with f3a4bf85136adb462a53d934d914b3782305723d7148a4eafc73df9fac6fdce4 not found: ID does not exist" Oct 31 14:57:08.300657 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.300582 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k"] Oct 31 14:57:08.306006 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.305984 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ee109dd4-77b76d56b5-4mb7k"] Oct 31 14:57:08.896067 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:08.896039 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d4db21-1311-49ed-a1dc-f9a38960c4ce" path="/var/lib/kubelet/pods/55d4db21-1311-49ed-a1dc-f9a38960c4ce/volumes" Oct 31 14:57:09.966216 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:09.966180 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j"] Oct 31 14:57:09.966651 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:09.966395 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" containerID="cri-o://523bc1c82eee3372d259c0ba698f3d0cc1f217015ccd00f67b6f6dbe85713ccf" gracePeriod=30 Oct 31 14:57:11.133862 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:11.133832 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 14:57:14.006610 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:14.006578 4399 patch_prober.go:29] interesting pod/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:57:14.007026 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:14.006658 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:57:15.138377 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.138346 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8"] Oct 31 14:57:15.139240 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.139217 4399 cpu_manager.go:414] "containerMap: removing container" podUID="55d4db21-1311-49ed-a1dc-f9a38960c4ce" containerName="maas-api" Oct 31 14:57:15.139413 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.139242 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d4db21-1311-49ed-a1dc-f9a38960c4ce" containerName="maas-api" Oct 31 14:57:15.139413 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.139353 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d4db21-1311-49ed-a1dc-f9a38960c4ce" containerName="maas-api" Oct 31 14:57:15.149975 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.149954 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:15.193823 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.193792 4399 generic.go:356] "Generic (PLEG): container finished" podID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerID="523bc1c82eee3372d259c0ba698f3d0cc1f217015ccd00f67b6f6dbe85713ccf" exitCode=0 Oct 31 14:57:15.193964 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.193927 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" event={"ID":"185b05b1-78c0-42fe-ae36-8cb67dd406a5","Type":"ContainerDied","Data":"523bc1c82eee3372d259c0ba698f3d0cc1f217015ccd00f67b6f6dbe85713ccf"} Oct 31 14:57:15.194022 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.193985 4399 scope.go:122] "RemoveContainer" containerID="38b1b8705d38cf388f08e9711fb72c1edf41f99fbf8c36e7c652dc1fb4bb0403" Oct 31 14:57:15.208326 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.208289 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8"] Oct 31 14:57:15.265789 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.263999 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-57815b\"" Oct 31 14:57:15.265789 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.264170 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-57815b-dockercfg-qvhqq\"" Oct 31 14:57:15.279433 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.279379 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m55f8\" (UniqueName: \"kubernetes.io/projected/87b7090c-0029-4f6c-ac6c-bd4810338241-kube-api-access-m55f8\") pod \"payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8\" (UID: \"87b7090c-0029-4f6c-ac6c-bd4810338241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:15.279592 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.279457 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b7090c-0029-4f6c-ac6c-bd4810338241-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8\" (UID: \"87b7090c-0029-4f6c-ac6c-bd4810338241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:15.303089 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.303060 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:57:15.315663 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.315630 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz"] Oct 31 14:57:15.316127 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.316115 4399 cpu_manager.go:414] "containerMap: removing container" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" Oct 31 14:57:15.316172 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.316129 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" Oct 31 14:57:15.316172 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.316137 4399 cpu_manager.go:414] "containerMap: removing container" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" Oct 31 14:57:15.316172 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.316142 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" Oct 31 14:57:15.316258 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.316229 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" Oct 31 14:57:15.316258 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.316237 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" containerName="istio-proxy" Oct 31 14:57:15.319367 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.319353 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:15.369748 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.369693 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz"] Oct 31 14:57:15.380288 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.380255 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m55f8\" (UniqueName: \"kubernetes.io/projected/87b7090c-0029-4f6c-ac6c-bd4810338241-kube-api-access-m55f8\") pod \"payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8\" (UID: \"87b7090c-0029-4f6c-ac6c-bd4810338241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:15.380473 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.380342 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b7090c-0029-4f6c-ac6c-bd4810338241-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8\" (UID: \"87b7090c-0029-4f6c-ac6c-bd4810338241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:15.381136 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.381106 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b7090c-0029-4f6c-ac6c-bd4810338241-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8\" (UID: \"87b7090c-0029-4f6c-ac6c-bd4810338241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:15.437595 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.437563 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m55f8\" (UniqueName: \"kubernetes.io/projected/87b7090c-0029-4f6c-ac6c-bd4810338241-kube-api-access-m55f8\") pod \"payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8\" (UID: \"87b7090c-0029-4f6c-ac6c-bd4810338241\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:15.481097 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481065 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istiod-ca-cert\") pod \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " Oct 31 14:57:15.481271 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481116 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-socket\") pod \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " Oct 31 14:57:15.481271 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481154 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-podinfo\") pod \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " Oct 31 14:57:15.481271 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481221 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-credential-socket\") pod \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " Oct 31 14:57:15.481426 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481306 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-token\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-token\") pod \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " Oct 31 14:57:15.481426 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481338 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-envoy\") pod \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " Oct 31 14:57:15.481426 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481402 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-kube-api-access-2jxxt\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-kube-api-access-2jxxt\") pod \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " Oct 31 14:57:15.481626 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481428 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-certs\") pod \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " Oct 31 14:57:15.481626 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481473 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-data\") pod \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\" (UID: \"185b05b1-78c0-42fe-ae36-8cb67dd406a5\") " Oct 31 14:57:15.481626 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481545 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-socket" pod "185b05b1-78c0-42fe-ae36-8cb67dd406a5" (UID: "185b05b1-78c0-42fe-ae36-8cb67dd406a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:15.481626 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481573 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-credential-socket" pod "185b05b1-78c0-42fe-ae36-8cb67dd406a5" (UID: "185b05b1-78c0-42fe-ae36-8cb67dd406a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:15.481867 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481797 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jcvf\" (UniqueName: \"kubernetes.io/projected/aad61f6f-8703-413b-8d9e-7538c8804de8-kube-api-access-2jcvf\") pod \"payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz\" (UID: \"aad61f6f-8703-413b-8d9e-7538c8804de8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:15.481867 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481855 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad61f6f-8703-413b-8d9e-7538c8804de8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz\" (UID: \"aad61f6f-8703-413b-8d9e-7538c8804de8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:15.481973 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481955 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:15.482025 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481972 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:15.482025 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.481980 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-certs" pod "185b05b1-78c0-42fe-ae36-8cb67dd406a5" (UID: "185b05b1-78c0-42fe-ae36-8cb67dd406a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:15.482384 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.482307 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-data" pod "185b05b1-78c0-42fe-ae36-8cb67dd406a5" (UID: "185b05b1-78c0-42fe-ae36-8cb67dd406a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:15.482827 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.482786 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istiod-ca-cert" pod "185b05b1-78c0-42fe-ae36-8cb67dd406a5" (UID: "185b05b1-78c0-42fe-ae36-8cb67dd406a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:57:15.484185 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.484006 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:15.484376 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.484307 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-envoy" pod "185b05b1-78c0-42fe-ae36-8cb67dd406a5" (UID: "185b05b1-78c0-42fe-ae36-8cb67dd406a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:57:15.484376 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.484347 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-podinfo" pod "185b05b1-78c0-42fe-ae36-8cb67dd406a5" (UID: "185b05b1-78c0-42fe-ae36-8cb67dd406a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:57:15.484376 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.484357 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-token" pod "185b05b1-78c0-42fe-ae36-8cb67dd406a5" (UID: "185b05b1-78c0-42fe-ae36-8cb67dd406a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:57:15.484590 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.484566 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-kube-api-access-2jxxt" pod "185b05b1-78c0-42fe-ae36-8cb67dd406a5" (UID: "185b05b1-78c0-42fe-ae36-8cb67dd406a5"). InnerVolumeSpecName "kube-api-access-2jxxt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:57:15.582685 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.582641 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jcvf\" (UniqueName: \"kubernetes.io/projected/aad61f6f-8703-413b-8d9e-7538c8804de8-kube-api-access-2jcvf\") pod \"payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz\" (UID: \"aad61f6f-8703-413b-8d9e-7538c8804de8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:15.582904 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.582765 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad61f6f-8703-413b-8d9e-7538c8804de8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz\" (UID: \"aad61f6f-8703-413b-8d9e-7538c8804de8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:15.582904 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.582894 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jxxt\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-kube-api-access-2jxxt\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:15.583024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.582915 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:15.583024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.582928 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:15.583024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.582943 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:15.583024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.582956 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:15.583024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.582969 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:15.583024 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.582982 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/185b05b1-78c0-42fe-ae36-8cb67dd406a5-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 14:57:15.583910 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.583886 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad61f6f-8703-413b-8d9e-7538c8804de8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz\" (UID: \"aad61f6f-8703-413b-8d9e-7538c8804de8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:15.640630 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.640600 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jcvf\" (UniqueName: \"kubernetes.io/projected/aad61f6f-8703-413b-8d9e-7538c8804de8-kube-api-access-2jcvf\") pod \"payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz\" (UID: \"aad61f6f-8703-413b-8d9e-7538c8804de8\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:15.875520 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.875441 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8"] Oct 31 14:57:15.877853 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:57:15.877825 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87b7090c_0029_4f6c_ac6c_bd4810338241.slice/crio-36dc044304828ca96aba0fd580276fb420da7ce057f96f78a2d24fde7974b014 WatchSource:0}: Error finding container 36dc044304828ca96aba0fd580276fb420da7ce057f96f78a2d24fde7974b014: Status 404 returned error can't find the container with id 36dc044304828ca96aba0fd580276fb420da7ce057f96f78a2d24fde7974b014 Oct 31 14:57:15.932817 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:15.932791 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:16.204256 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.203687 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" event={"ID":"87b7090c-0029-4f6c-ac6c-bd4810338241","Type":"ContainerStarted","Data":"1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1"} Oct 31 14:57:16.204256 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.203787 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:16.204256 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.203812 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" event={"ID":"87b7090c-0029-4f6c-ac6c-bd4810338241","Type":"ContainerStarted","Data":"36dc044304828ca96aba0fd580276fb420da7ce057f96f78a2d24fde7974b014"} Oct 31 14:57:16.205332 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.205313 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" Oct 31 14:57:16.205444 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.205337 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j" event={"ID":"185b05b1-78c0-42fe-ae36-8cb67dd406a5","Type":"ContainerDied","Data":"6429db260e094a42acbd62787ec87551b5fde800f650062145de83d2dadc19fc"} Oct 31 14:57:16.205444 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.205381 4399 scope.go:122] "RemoveContainer" containerID="523bc1c82eee3372d259c0ba698f3d0cc1f217015ccd00f67b6f6dbe85713ccf" Oct 31 14:57:16.333488 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.333428 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" podStartSLOduration=1.333411038 podStartE2EDuration="1.333411038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:16.332113884 +0000 UTC m=+679.543355312" watchObservedRunningTime="2026-10-31 14:57:16.333411038 +0000 UTC m=+679.544652450" Oct 31 14:57:16.338225 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:57:16.338201 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaad61f6f_8703_413b_8d9e_7538c8804de8.slice/crio-a35f4ea96134fde9cf146e7184ae5718a71a044eda46f9863ca42e79fe234b7c WatchSource:0}: Error finding container a35f4ea96134fde9cf146e7184ae5718a71a044eda46f9863ca42e79fe234b7c: Status 404 returned error can't find the container with id a35f4ea96134fde9cf146e7184ae5718a71a044eda46f9863ca42e79fe234b7c Oct 31 14:57:16.339263 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.339184 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz"] Oct 31 14:57:16.450117 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.450086 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j"] Oct 31 14:57:16.456429 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.456348 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ee109dd4-openshift-default-b8df96cc7-wsp7j"] Oct 31 14:57:16.898225 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:16.898151 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="185b05b1-78c0-42fe-ae36-8cb67dd406a5" path="/var/lib/kubelet/pods/185b05b1-78c0-42fe-ae36-8cb67dd406a5/volumes" Oct 31 14:57:17.213258 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:17.213218 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" event={"ID":"aad61f6f-8703-413b-8d9e-7538c8804de8","Type":"ContainerStarted","Data":"8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb"} Oct 31 14:57:17.213773 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:17.213285 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" event={"ID":"aad61f6f-8703-413b-8d9e-7538c8804de8","Type":"ContainerStarted","Data":"a35f4ea96134fde9cf146e7184ae5718a71a044eda46f9863ca42e79fe234b7c"} Oct 31 14:57:17.213773 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:17.213313 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:17.328387 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:17.328322 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" podStartSLOduration=2.328296155 podStartE2EDuration="2.328296155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:17.327096251 +0000 UTC m=+680.538337664" watchObservedRunningTime="2026-10-31 14:57:17.328296155 +0000 UTC m=+680.539537568" Oct 31 14:57:19.992740 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:19.992706 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs"] Oct 31 14:57:19.996478 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:19.996462 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:20.030102 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.030075 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfbe731-fa58-4f55-954a-daa795b4ec31-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs\" (UID: \"acfbe731-fa58-4f55-954a-daa795b4ec31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:20.030222 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.030124 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mzm2\" (UniqueName: \"kubernetes.io/projected/acfbe731-fa58-4f55-954a-daa795b4ec31-kube-api-access-2mzm2\") pod \"payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs\" (UID: \"acfbe731-fa58-4f55-954a-daa795b4ec31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:20.053763 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.053722 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs"] Oct 31 14:57:20.099408 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.099252 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-fb2348\"" Oct 31 14:57:20.099879 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.099856 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-fb2348-dockercfg-xzbvs\"" Oct 31 14:57:20.131122 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.131095 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mzm2\" (UniqueName: \"kubernetes.io/projected/acfbe731-fa58-4f55-954a-daa795b4ec31-kube-api-access-2mzm2\") pod \"payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs\" (UID: \"acfbe731-fa58-4f55-954a-daa795b4ec31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:20.131269 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.131213 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfbe731-fa58-4f55-954a-daa795b4ec31-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs\" (UID: \"acfbe731-fa58-4f55-954a-daa795b4ec31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:20.131844 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.131825 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfbe731-fa58-4f55-954a-daa795b4ec31-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs\" (UID: \"acfbe731-fa58-4f55-954a-daa795b4ec31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:20.156267 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.156244 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn"] Oct 31 14:57:20.160055 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.160028 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:20.197457 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.197423 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mzm2\" (UniqueName: \"kubernetes.io/projected/acfbe731-fa58-4f55-954a-daa795b4ec31-kube-api-access-2mzm2\") pod \"payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs\" (UID: \"acfbe731-fa58-4f55-954a-daa795b4ec31\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:20.215845 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.215815 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn"] Oct 31 14:57:20.231913 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.231883 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee41d29-f36e-4c04-93dd-c5b96ad49396-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn\" (UID: \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:20.232050 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.232031 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrx2s\" (UniqueName: \"kubernetes.io/projected/7ee41d29-f36e-4c04-93dd-c5b96ad49396-kube-api-access-jrx2s\") pod \"payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn\" (UID: \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:20.232341 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.232316 4399 generic.go:356] "Generic (PLEG): container finished" podID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerID="f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0" exitCode=0 Oct 31 14:57:20.232407 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.232374 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" event={"ID":"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300","Type":"ContainerDied","Data":"f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0"} Oct 31 14:57:20.232837 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.232820 4399 scope.go:122] "RemoveContainer" containerID="f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0" Oct 31 14:57:20.234039 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.234009 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:57:20.234143 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.234086 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:57:20.234143 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.234133 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:57:20.309791 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.309766 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:20.333346 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.333312 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee41d29-f36e-4c04-93dd-c5b96ad49396-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn\" (UID: \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:20.333441 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.333407 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrx2s\" (UniqueName: \"kubernetes.io/projected/7ee41d29-f36e-4c04-93dd-c5b96ad49396-kube-api-access-jrx2s\") pod \"payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn\" (UID: \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:20.334483 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.334457 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee41d29-f36e-4c04-93dd-c5b96ad49396-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn\" (UID: \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:20.397511 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.397478 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrx2s\" (UniqueName: \"kubernetes.io/projected/7ee41d29-f36e-4c04-93dd-c5b96ad49396-kube-api-access-jrx2s\") pod \"payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn\" (UID: \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:20.473203 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.473172 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:20.686740 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.686719 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs"] Oct 31 14:57:20.688103 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:57:20.688073 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacfbe731_fa58_4f55_954a_daa795b4ec31.slice/crio-c7a793715045b78502607efba93d5577e052a7fd066095467c93b9caedb6bc4f WatchSource:0}: Error finding container c7a793715045b78502607efba93d5577e052a7fd066095467c93b9caedb6bc4f: Status 404 returned error can't find the container with id c7a793715045b78502607efba93d5577e052a7fd066095467c93b9caedb6bc4f Oct 31 14:57:20.870999 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:20.869851 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn"] Oct 31 14:57:20.873086 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:57:20.873055 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ee41d29_f36e_4c04_93dd_c5b96ad49396.slice/crio-b572de563fc88ee9b1a8d3bdd8e2a52a69d583d551aeb705e091c9640df44cb4 WatchSource:0}: Error finding container b572de563fc88ee9b1a8d3bdd8e2a52a69d583d551aeb705e091c9640df44cb4: Status 404 returned error can't find the container with id b572de563fc88ee9b1a8d3bdd8e2a52a69d583d551aeb705e091c9640df44cb4 Oct 31 14:57:21.239632 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.239593 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" event={"ID":"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300","Type":"ContainerStarted","Data":"aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa"} Oct 31 14:57:21.259723 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.254629 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" event={"ID":"7ee41d29-f36e-4c04-93dd-c5b96ad49396","Type":"ContainerStarted","Data":"814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13"} Oct 31 14:57:21.259723 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.254725 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" event={"ID":"7ee41d29-f36e-4c04-93dd-c5b96ad49396","Type":"ContainerStarted","Data":"b572de563fc88ee9b1a8d3bdd8e2a52a69d583d551aeb705e091c9640df44cb4"} Oct 31 14:57:21.259723 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.255415 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:21.259723 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.257455 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" event={"ID":"acfbe731-fa58-4f55-954a-daa795b4ec31","Type":"ContainerStarted","Data":"f5045ce07a2e02e32792fb7bb111d8a28504f26208be13e53a7c8ae0847d6c08"} Oct 31 14:57:21.259723 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.257498 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" event={"ID":"acfbe731-fa58-4f55-954a-daa795b4ec31","Type":"ContainerStarted","Data":"c7a793715045b78502607efba93d5577e052a7fd066095467c93b9caedb6bc4f"} Oct 31 14:57:21.259723 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.258101 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:21.484590 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.484536 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" podStartSLOduration=1.484521744 podStartE2EDuration="1.484521744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:21.481593245 +0000 UTC m=+684.692834657" watchObservedRunningTime="2026-10-31 14:57:21.484521744 +0000 UTC m=+684.695763153" Oct 31 14:57:21.594624 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.594498 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" podStartSLOduration=2.594479013 podStartE2EDuration="2.594479013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:21.593765459 +0000 UTC m=+684.805006870" watchObservedRunningTime="2026-10-31 14:57:21.594479013 +0000 UTC m=+684.805720425" Oct 31 14:57:21.986912 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.986877 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:57:21.987091 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.987033 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-ht48f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 14:57:21.987091 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:21.987079 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 14:57:22.986583 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:22.986551 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-ht48f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 14:57:22.987009 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:22.986601 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 14:57:23.987060 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:23.987025 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-ht48f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 14:57:23.987523 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:23.987085 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 14:57:24.987091 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:24.987062 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-ht48f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 14:57:24.987575 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:24.987119 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 14:57:25.986570 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:25.986534 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-ht48f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 31 14:57:25.986750 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:25.986583 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 31 14:57:26.206830 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:57:26.206795 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:57:26.990060 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:26.990034 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:57:26.990309 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:26.990295 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:57:26.990982 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:26.990965 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 14:57:27.217567 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:27.217543 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 14:57:28.222061 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:28.222034 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 14:57:33.279149 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:33.279121 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 14:57:33.279588 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:33.279188 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 14:57:54.161681 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.161651 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j"] Oct 31 14:57:54.165832 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.165810 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.191458 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.191435 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d10edc-610e-4f43-8676-ab2e687db74b-tls-certs\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.191628 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.191464 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkbh4\" (UniqueName: \"kubernetes.io/projected/b5d10edc-610e-4f43-8676-ab2e687db74b-kube-api-access-mkbh4\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.191628 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.191594 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-home\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.191795 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.191673 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-model-cache\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.191795 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.191714 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-tmp-dir\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.191795 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.191738 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-dshm\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.271496 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.271470 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"kube-root-ca.crt\"" Oct 31 14:57:54.271664 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.271565 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"default-dockercfg-7gwx9\"" Oct 31 14:57:54.271664 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.271574 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"openshift-service-ca.crt\"" Oct 31 14:57:54.271803 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.271679 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"simulator-w3-190693-kserve-self-signed-certs\"" Oct 31 14:57:54.273913 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.273893 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j"] Oct 31 14:57:54.292308 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.292281 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-dshm\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.292421 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.292371 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d10edc-610e-4f43-8676-ab2e687db74b-tls-certs\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.292421 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.292402 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkbh4\" (UniqueName: \"kubernetes.io/projected/b5d10edc-610e-4f43-8676-ab2e687db74b-kube-api-access-mkbh4\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.292518 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.292442 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-home\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.292518 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.292484 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-model-cache\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.292518 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.292500 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-tmp-dir\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.292813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.292793 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-home\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.292930 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.292861 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-model-cache\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.292930 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.292917 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-tmp-dir\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.294642 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.294614 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-dshm\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.294915 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.294893 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d10edc-610e-4f43-8676-ab2e687db74b-tls-certs\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.349363 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.349332 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkbh4\" (UniqueName: \"kubernetes.io/projected/b5d10edc-610e-4f43-8676-ab2e687db74b-kube-api-access-mkbh4\") pod \"simulator-w3-190693-kserve-78c4b4c854-gsn6j\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.478602 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.478572 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:54.855667 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:54.855645 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j"] Oct 31 14:57:54.857769 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:57:54.857737 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5d10edc_610e_4f43_8676_ab2e687db74b.slice/crio-b73607791a9a6b6f6cccd6d9722e4e45eac306d8e2c437e6d458dc692378a4ca WatchSource:0}: Error finding container b73607791a9a6b6f6cccd6d9722e4e45eac306d8e2c437e6d458dc692378a4ca: Status 404 returned error can't find the container with id b73607791a9a6b6f6cccd6d9722e4e45eac306d8e2c437e6d458dc692378a4ca Oct 31 14:57:55.449222 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:55.449181 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" event={"ID":"b5d10edc-610e-4f43-8676-ab2e687db74b","Type":"ContainerStarted","Data":"a3bb92da2f0d4008f9d65d51c51771f3df04946049241dc6321226f0c23d35af"} Oct 31 14:57:55.449222 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:55.449225 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" event={"ID":"b5d10edc-610e-4f43-8676-ab2e687db74b","Type":"ContainerStarted","Data":"b73607791a9a6b6f6cccd6d9722e4e45eac306d8e2c437e6d458dc692378a4ca"} Oct 31 14:57:55.479507 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:55.479481 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:57:55.563815 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:57:55.563772 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podStartSLOduration=1.563757656 podStartE2EDuration="1.563757656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:57:55.563003151 +0000 UTC m=+718.774244569" watchObservedRunningTime="2026-10-31 14:57:55.563757656 +0000 UTC m=+718.774999103" Oct 31 14:57:56.206199 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:57:56.206166 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:58:04.479120 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:04.479080 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:58:04.491755 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:04.491731 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:58:04.506112 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:04.506090 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 14:58:26.205843 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:58:26.205795 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:58:45.086234 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.086146 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9"] Oct 31 14:58:45.090323 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.090301 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.196221 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.196191 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"openshift-service-ca.crt\"" Oct 31 14:58:45.196428 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.196229 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"kube-root-ca.crt\"" Oct 31 14:58:45.196428 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.196333 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"premium-w0-56bc96-kserve-self-signed-certs\"" Oct 31 14:58:45.196892 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.196873 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-56bc96\"/\"default-dockercfg-zxkjx\"" Oct 31 14:58:45.199095 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.199065 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9"] Oct 31 14:58:45.211478 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.211454 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-dshm\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.211623 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.211496 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plsf5\" (UniqueName: \"kubernetes.io/projected/79e5d115-2489-4b78-a250-6f3ef934b12b-kube-api-access-plsf5\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.211715 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.211643 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-home\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.211784 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.211731 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-tmp-dir\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.211861 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.211847 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-model-cache\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.211917 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.211880 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79e5d115-2489-4b78-a250-6f3ef934b12b-tls-certs\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.312496 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.312452 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-model-cache\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.312496 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.312501 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79e5d115-2489-4b78-a250-6f3ef934b12b-tls-certs\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.312813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.312563 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-dshm\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.312813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.312605 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plsf5\" (UniqueName: \"kubernetes.io/projected/79e5d115-2489-4b78-a250-6f3ef934b12b-kube-api-access-plsf5\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.312813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.312656 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-home\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.312813 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.312768 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-tmp-dir\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.313047 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.312947 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-model-cache\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.313111 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.313047 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-home\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.313111 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.313100 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-tmp-dir\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.315141 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.315118 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-dshm\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.315491 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.315461 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79e5d115-2489-4b78-a250-6f3ef934b12b-tls-certs\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.369145 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.369070 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plsf5\" (UniqueName: \"kubernetes.io/projected/79e5d115-2489-4b78-a250-6f3ef934b12b-kube-api-access-plsf5\") pod \"premium-w0-56bc96-kserve-6577499bc9-45dh9\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.402136 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.402108 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:45.774764 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:45.774736 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9"] Oct 31 14:58:45.777468 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:58:45.777441 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79e5d115_2489_4b78_a250_6f3ef934b12b.slice/crio-5285a65f85183acf4fd3c55938e69286fc111849826c9ecdfc3e6b0b031072ae WatchSource:0}: Error finding container 5285a65f85183acf4fd3c55938e69286fc111849826c9ecdfc3e6b0b031072ae: Status 404 returned error can't find the container with id 5285a65f85183acf4fd3c55938e69286fc111849826c9ecdfc3e6b0b031072ae Oct 31 14:58:46.717373 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:46.717338 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" event={"ID":"79e5d115-2489-4b78-a250-6f3ef934b12b","Type":"ContainerStarted","Data":"9445dd8b7f12e145b48caca0d284f26bcd767ddc19a6555589f2c6584e688a0b"} Oct 31 14:58:46.717373 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:46.717379 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" event={"ID":"79e5d115-2489-4b78-a250-6f3ef934b12b","Type":"ContainerStarted","Data":"5285a65f85183acf4fd3c55938e69286fc111849826c9ecdfc3e6b0b031072ae"} Oct 31 14:58:46.831127 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:46.831077 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podStartSLOduration=1.831063871 podStartE2EDuration="1.831063871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:58:46.830003361 +0000 UTC m=+770.041244773" watchObservedRunningTime="2026-10-31 14:58:46.831063871 +0000 UTC m=+770.042305282" Oct 31 14:58:47.402928 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:47.402897 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:55.402435 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:55.402402 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:55.415093 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:55.415067 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:55.778744 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:58:55.778717 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 14:58:56.205601 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:58:56.205572 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:59:02.479959 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.479925 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf"] Oct 31 14:59:02.484018 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.484002 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.496087 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.496059 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.496229 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.496109 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.496229 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.496187 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.496229 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.496218 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.496412 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.496255 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.496412 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.496371 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.496526 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.496412 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946w6\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-kube-api-access-946w6\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.496526 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.496483 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.496526 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.496517 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.536407 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.536387 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-029d26-openshift-default-dockercfg-lxm75\"" Oct 31 14:59:02.590391 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.590365 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf"] Oct 31 14:59:02.597302 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597280 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597405 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597314 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597405 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597347 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597405 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597380 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597542 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597406 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597542 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597479 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597542 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597507 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946w6\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-kube-api-access-946w6\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597686 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597566 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597686 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597598 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597834 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597814 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.597897 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597855 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.598507 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.597888 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.598507 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.598218 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.598660 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.598603 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.600763 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.600741 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.601108 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.601080 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.654514 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.654488 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.654514 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.654505 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-946w6\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-kube-api-access-946w6\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:02.798672 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:02.798597 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:03.224928 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:03.224905 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf"] Oct 31 14:59:03.226573 ip-10-0-137-160 kubenswrapper[4399]: W1031 14:59:03.226545 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17e6b5a0_a02a_4d6a_b190_7e1f7176fb3e.slice/crio-f56c467e3755343162a842961f9e71eed77a3b6c7bfa760e0470e2e3a77e8aef WatchSource:0}: Error finding container f56c467e3755343162a842961f9e71eed77a3b6c7bfa760e0470e2e3a77e8aef: Status 404 returned error can't find the container with id f56c467e3755343162a842961f9e71eed77a3b6c7bfa760e0470e2e3a77e8aef Oct 31 14:59:03.228740 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:03.228680 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:59:03.228789 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:03.228775 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:59:03.228830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:03.228805 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:59:03.815061 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:03.815023 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" event={"ID":"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e","Type":"ContainerStarted","Data":"3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d"} Oct 31 14:59:03.815452 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:03.815077 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" event={"ID":"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e","Type":"ContainerStarted","Data":"f56c467e3755343162a842961f9e71eed77a3b6c7bfa760e0470e2e3a77e8aef"} Oct 31 14:59:03.980434 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:03.980374 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podStartSLOduration=1.980359685 podStartE2EDuration="1.980359685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:59:03.979592623 +0000 UTC m=+787.190834035" watchObservedRunningTime="2026-10-31 14:59:03.980359685 +0000 UTC m=+787.191601096" Oct 31 14:59:04.799410 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:04.799378 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:04.800879 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:04.800854 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 14:59:04.800993 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:04.800905 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 14:59:05.799349 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:05.799312 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 14:59:05.799837 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:05.799370 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 14:59:06.799061 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:06.799024 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 14:59:06.799246 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:06.799072 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 14:59:07.799450 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:07.799413 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 14:59:07.799845 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:07.799476 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 14:59:08.799301 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:08.799262 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 14:59:08.799520 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:08.799311 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 14:59:09.798947 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:09.798909 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Oct 31 14:59:09.799154 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:09.798969 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Oct 31 14:59:10.803830 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:10.803803 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:10.804268 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:10.804187 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:10.804880 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:10.804864 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 14:59:26.205902 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:59:26.205870 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:59:50.342633 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.342598 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk"] Oct 31 14:59:50.347510 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.347486 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.413671 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.413644 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-56bc96-openshift-default-dockercfg-5qxbd\"" Oct 31 14:59:50.441160 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.441127 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.441281 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.441177 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.441281 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.441243 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.441350 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.441283 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.441404 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.441384 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.441452 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.441440 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4907085c-e7cb-4cec-a720-4fa41cf6060c-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.441507 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.441490 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5lcc\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-kube-api-access-x5lcc\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.441548 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.441536 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.441595 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.441580 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.457052 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.457023 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk"] Oct 31 14:59:50.542528 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.542496 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.542684 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.542537 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.542684 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.542592 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.542684 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.542614 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.542684 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.542629 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.542684 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.542664 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.542954 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.542723 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.542954 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.542773 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4907085c-e7cb-4cec-a720-4fa41cf6060c-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.542954 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.542819 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5lcc\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-kube-api-access-x5lcc\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.543112 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.543072 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.543112 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.543091 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.543515 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.543143 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.543861 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.543831 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.544603 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.544335 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4907085c-e7cb-4cec-a720-4fa41cf6060c-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.552212 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.552191 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.552805 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.552782 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.599659 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.599593 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5lcc\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-kube-api-access-x5lcc\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.599803 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.599771 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:50.662839 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:50.662809 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:51.091770 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:51.091746 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk"] Oct 31 14:59:51.098154 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:51.098125 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:59:51.098264 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:51.098188 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:59:51.098264 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:51.098219 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 14:59:51.104488 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:51.104461 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" event={"ID":"4907085c-e7cb-4cec-a720-4fa41cf6060c","Type":"ContainerStarted","Data":"c94d2c68342d8de0c283beb92768e3dab622faca4df82c512904ea5b2c0c35d7"} Oct 31 14:59:52.111613 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:52.111578 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" event={"ID":"4907085c-e7cb-4cec-a720-4fa41cf6060c","Type":"ContainerStarted","Data":"dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b"} Oct 31 14:59:52.328265 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:52.328207 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" podStartSLOduration=2.328191802 podStartE2EDuration="2.328191802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:59:52.276916969 +0000 UTC m=+835.488158381" watchObservedRunningTime="2026-10-31 14:59:52.328191802 +0000 UTC m=+835.539433213" Oct 31 14:59:52.666912 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:52.666874 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:52.668677 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:52.668649 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 14:59:52.668843 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:52.668732 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 14:59:53.665152 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:53.665116 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 14:59:53.665768 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:53.665723 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 14:59:54.666141 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:54.666110 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 14:59:54.666504 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:54.666164 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 14:59:55.664181 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:55.664146 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 14:59:55.664484 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:55.664455 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 14:59:56.212975 ip-10-0-137-160 kubenswrapper[4399]: E1031 14:59:56.212935 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 14:59:56.666221 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:56.666139 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 14:59:56.666221 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:56.666197 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 14:59:57.664165 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:57.664122 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Oct 31 14:59:57.665027 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:57.664181 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Oct 31 14:59:58.667771 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:58.667741 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:58.668233 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:58.668124 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 14:59:58.668923 ip-10-0-137-160 kubenswrapper[4399]: I1031 14:59:58.668908 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 15:00:00.166970 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.166935 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9"] Oct 31 15:00:00.171246 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.171228 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:00.222425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.222403 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:00:00.272506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.272476 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmx9\" (UniqueName: \"kubernetes.io/projected/f905df74-24ab-43d1-9641-7ec0677708cb-kube-api-access-rnmx9\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9\" (UID: \"f905df74-24ab-43d1-9641-7ec0677708cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:00.272674 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.272662 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f905df74-24ab-43d1-9641-7ec0677708cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9\" (UID: \"f905df74-24ab-43d1-9641-7ec0677708cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:00.275964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.275940 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9"] Oct 31 15:00:00.373606 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.373573 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmx9\" (UniqueName: \"kubernetes.io/projected/f905df74-24ab-43d1-9641-7ec0677708cb-kube-api-access-rnmx9\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9\" (UID: \"f905df74-24ab-43d1-9641-7ec0677708cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:00.373789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.373741 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f905df74-24ab-43d1-9641-7ec0677708cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9\" (UID: \"f905df74-24ab-43d1-9641-7ec0677708cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:00.374381 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.374359 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f905df74-24ab-43d1-9641-7ec0677708cb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9\" (UID: \"f905df74-24ab-43d1-9641-7ec0677708cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:00.430053 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.429985 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmx9\" (UniqueName: \"kubernetes.io/projected/f905df74-24ab-43d1-9641-7ec0677708cb-kube-api-access-rnmx9\") pod \"maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9\" (UID: \"f905df74-24ab-43d1-9641-7ec0677708cb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:00.513094 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.513065 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:00.888663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:00.888635 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9"] Oct 31 15:00:00.891962 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:00:00.891938 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf905df74_24ab_43d1_9641_7ec0677708cb.slice/crio-e7e9f5da4c288055a83c25df597c80ef336dbeea0de0366076ca9798a8a9d1d6 WatchSource:0}: Error finding container e7e9f5da4c288055a83c25df597c80ef336dbeea0de0366076ca9798a8a9d1d6: Status 404 returned error can't find the container with id e7e9f5da4c288055a83c25df597c80ef336dbeea0de0366076ca9798a8a9d1d6 Oct 31 15:00:01.175277 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:01.175196 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" event={"ID":"f905df74-24ab-43d1-9641-7ec0677708cb","Type":"ContainerStarted","Data":"e7e9f5da4c288055a83c25df597c80ef336dbeea0de0366076ca9798a8a9d1d6"} Oct 31 15:00:02.271223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.271190 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd"] Oct 31 15:00:02.276836 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.276818 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.330135 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.330100 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd"] Oct 31 15:00:02.376434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.376403 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:00:02.394034 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.394002 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e8e9c4b-6zcjd\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.394168 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.394063 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzfhn\" (UniqueName: \"kubernetes.io/projected/960dda34-9d5f-4fde-aba0-ca81b683712e-kube-api-access-fzfhn\") pod \"maas-api-revoke-sub-4e8e9c4b-6zcjd\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.394234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.394167 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e8e9c4b-6zcjd\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.495317 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.495282 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e8e9c4b-6zcjd\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.495517 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.495387 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e8e9c4b-6zcjd\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.495517 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.495429 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzfhn\" (UniqueName: \"kubernetes.io/projected/960dda34-9d5f-4fde-aba0-ca81b683712e-kube-api-access-fzfhn\") pod \"maas-api-revoke-sub-4e8e9c4b-6zcjd\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.496073 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.496040 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e8e9c4b-6zcjd\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.498001 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.497981 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e8e9c4b-6zcjd\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.552050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.551982 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzfhn\" (UniqueName: \"kubernetes.io/projected/960dda34-9d5f-4fde-aba0-ca81b683712e-kube-api-access-fzfhn\") pod \"maas-api-revoke-sub-4e8e9c4b-6zcjd\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.590839 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.590813 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:02.961396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:02.961373 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd"] Oct 31 15:00:02.964101 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:00:02.964074 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960dda34_9d5f_4fde_aba0_ca81b683712e.slice/crio-f58748f936cd8e7c888f86f5452b16c77d44a6fe28eaa00f21ba1cd2079dd2ec WatchSource:0}: Error finding container f58748f936cd8e7c888f86f5452b16c77d44a6fe28eaa00f21ba1cd2079dd2ec: Status 404 returned error can't find the container with id f58748f936cd8e7c888f86f5452b16c77d44a6fe28eaa00f21ba1cd2079dd2ec Oct 31 15:00:03.186770 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:03.186724 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" event={"ID":"960dda34-9d5f-4fde-aba0-ca81b683712e","Type":"ContainerStarted","Data":"f58748f936cd8e7c888f86f5452b16c77d44a6fe28eaa00f21ba1cd2079dd2ec"} Oct 31 15:00:09.228171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:09.228142 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" event={"ID":"f905df74-24ab-43d1-9641-7ec0677708cb","Type":"ContainerStarted","Data":"7c0fac3788301a37737ff26827377662a62b37ba911e25b7e06cb0eb46dab920"} Oct 31 15:00:09.340661 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:09.340560 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" podStartSLOduration=1.468083871 podStartE2EDuration="9.340542314s" totalImagesPullingTime="7.872458443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:09.339865638 +0000 UTC m=+852.551107043" watchObservedRunningTime="2026-10-31 15:00:09.340542314 +0000 UTC m=+852.551783727" Oct 31 15:00:10.235290 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:10.235262 4399 generic.go:356] "Generic (PLEG): container finished" podID="960dda34-9d5f-4fde-aba0-ca81b683712e" containerID="b374ca6b5bd6b32a97e0c8efe34216cb891a09be0b8cd53ba6516f84838bfab1" exitCode=0 Oct 31 15:00:10.235679 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:10.235356 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" event={"ID":"960dda34-9d5f-4fde-aba0-ca81b683712e","Type":"ContainerDied","Data":"b374ca6b5bd6b32a97e0c8efe34216cb891a09be0b8cd53ba6516f84838bfab1"} Oct 31 15:00:10.236913 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:10.236890 4399 generic.go:356] "Generic (PLEG): container finished" podID="f905df74-24ab-43d1-9641-7ec0677708cb" containerID="7c0fac3788301a37737ff26827377662a62b37ba911e25b7e06cb0eb46dab920" exitCode=60 Oct 31 15:00:10.237023 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:10.236973 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" event={"ID":"f905df74-24ab-43d1-9641-7ec0677708cb","Type":"ContainerDied","Data":"7c0fac3788301a37737ff26827377662a62b37ba911e25b7e06cb0eb46dab920"} Oct 31 15:00:10.237150 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:10.237138 4399 scope.go:122] "RemoveContainer" containerID="7c0fac3788301a37737ff26827377662a62b37ba911e25b7e06cb0eb46dab920" Oct 31 15:00:11.245200 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.245156 4399 generic.go:356] "Generic (PLEG): container finished" podID="f905df74-24ab-43d1-9641-7ec0677708cb" containerID="8eed6bbf77d43f4bd4589098d83f1e4fb4c1c488987b2f9b590d69687a6f8889" exitCode=60 Oct 31 15:00:11.245624 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.245226 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" event={"ID":"f905df74-24ab-43d1-9641-7ec0677708cb","Type":"ContainerDied","Data":"8eed6bbf77d43f4bd4589098d83f1e4fb4c1c488987b2f9b590d69687a6f8889"} Oct 31 15:00:11.245624 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.245309 4399 scope.go:122] "RemoveContainer" containerID="7c0fac3788301a37737ff26827377662a62b37ba911e25b7e06cb0eb46dab920" Oct 31 15:00:11.245624 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.245555 4399 scope.go:122] "RemoveContainer" containerID="8eed6bbf77d43f4bd4589098d83f1e4fb4c1c488987b2f9b590d69687a6f8889" Oct 31 15:00:11.246165 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:00:11.245843 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9_odh-ai-gateway-infra(f905df74-24ab-43d1-9641-7ec0677708cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" Oct 31 15:00:11.393190 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.393135 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:11.415282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.415257 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-xks-ca\") pod \"960dda34-9d5f-4fde-aba0-ca81b683712e\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " Oct 31 15:00:11.415417 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.415324 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-openshift-ca\") pod \"960dda34-9d5f-4fde-aba0-ca81b683712e\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " Oct 31 15:00:11.415476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.415437 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960dda34-9d5f-4fde-aba0-ca81b683712e-kube-api-access-fzfhn\" (UniqueName: \"kubernetes.io/projected/960dda34-9d5f-4fde-aba0-ca81b683712e-kube-api-access-fzfhn\") pod \"960dda34-9d5f-4fde-aba0-ca81b683712e\" (UID: \"960dda34-9d5f-4fde-aba0-ca81b683712e\") " Oct 31 15:00:11.415689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.415666 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-openshift-ca" pod "960dda34-9d5f-4fde-aba0-ca81b683712e" (UID: "960dda34-9d5f-4fde-aba0-ca81b683712e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:00:11.415861 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.415843 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:11.417394 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.417375 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-xks-ca" pod "960dda34-9d5f-4fde-aba0-ca81b683712e" (UID: "960dda34-9d5f-4fde-aba0-ca81b683712e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:00:11.417481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.417471 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960dda34-9d5f-4fde-aba0-ca81b683712e-kube-api-access-fzfhn" pod "960dda34-9d5f-4fde-aba0-ca81b683712e" (UID: "960dda34-9d5f-4fde-aba0-ca81b683712e"). InnerVolumeSpecName "kube-api-access-fzfhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:00:11.517068 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.517041 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzfhn\" (UniqueName: \"kubernetes.io/projected/960dda34-9d5f-4fde-aba0-ca81b683712e-kube-api-access-fzfhn\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:11.517068 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:11.517065 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960dda34-9d5f-4fde-aba0-ca81b683712e-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:12.252137 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:12.252108 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" Oct 31 15:00:12.252596 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:12.252113 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd" event={"ID":"960dda34-9d5f-4fde-aba0-ca81b683712e","Type":"ContainerDied","Data":"f58748f936cd8e7c888f86f5452b16c77d44a6fe28eaa00f21ba1cd2079dd2ec"} Oct 31 15:00:12.252596 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:12.252225 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f58748f936cd8e7c888f86f5452b16c77d44a6fe28eaa00f21ba1cd2079dd2ec" Oct 31 15:00:12.253940 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:12.253924 4399 scope.go:122] "RemoveContainer" containerID="8eed6bbf77d43f4bd4589098d83f1e4fb4c1c488987b2f9b590d69687a6f8889" Oct 31 15:00:12.254141 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:00:12.254124 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9_odh-ai-gateway-infra(f905df74-24ab-43d1-9641-7ec0677708cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" Oct 31 15:00:22.893951 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:22.891825 4399 scope.go:122] "RemoveContainer" containerID="8eed6bbf77d43f4bd4589098d83f1e4fb4c1c488987b2f9b590d69687a6f8889" Oct 31 15:00:24.327805 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:24.327772 4399 generic.go:356] "Generic (PLEG): container finished" podID="f905df74-24ab-43d1-9641-7ec0677708cb" containerID="65ecd9a0f634785363e55e5eb042124c0f2d334a3adaf69b4503920eac8a9003" exitCode=60 Oct 31 15:00:24.328227 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:24.327840 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" event={"ID":"f905df74-24ab-43d1-9641-7ec0677708cb","Type":"ContainerDied","Data":"65ecd9a0f634785363e55e5eb042124c0f2d334a3adaf69b4503920eac8a9003"} Oct 31 15:00:24.328227 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:24.327904 4399 scope.go:122] "RemoveContainer" containerID="8eed6bbf77d43f4bd4589098d83f1e4fb4c1c488987b2f9b590d69687a6f8889" Oct 31 15:00:24.328365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:24.328264 4399 scope.go:122] "RemoveContainer" containerID="65ecd9a0f634785363e55e5eb042124c0f2d334a3adaf69b4503920eac8a9003" Oct 31 15:00:24.328621 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:00:24.328597 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9_odh-ai-gateway-infra(f905df74-24ab-43d1-9641-7ec0677708cb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" Oct 31 15:00:25.453164 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:25.453127 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9"] Oct 31 15:00:25.632433 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:25.632405 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:25.685503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:25.685473 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f905df74-24ab-43d1-9641-7ec0677708cb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f905df74-24ab-43d1-9641-7ec0677708cb-maas-api-service-ca\") pod \"f905df74-24ab-43d1-9641-7ec0677708cb\" (UID: \"f905df74-24ab-43d1-9641-7ec0677708cb\") " Oct 31 15:00:25.685625 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:25.685578 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f905df74-24ab-43d1-9641-7ec0677708cb-kube-api-access-rnmx9\" (UniqueName: \"kubernetes.io/projected/f905df74-24ab-43d1-9641-7ec0677708cb-kube-api-access-rnmx9\") pod \"f905df74-24ab-43d1-9641-7ec0677708cb\" (UID: \"f905df74-24ab-43d1-9641-7ec0677708cb\") " Oct 31 15:00:25.685831 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:25.685809 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f905df74-24ab-43d1-9641-7ec0677708cb-maas-api-service-ca" pod "f905df74-24ab-43d1-9641-7ec0677708cb" (UID: "f905df74-24ab-43d1-9641-7ec0677708cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:00:25.687642 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:25.687623 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f905df74-24ab-43d1-9641-7ec0677708cb-kube-api-access-rnmx9" pod "f905df74-24ab-43d1-9641-7ec0677708cb" (UID: "f905df74-24ab-43d1-9641-7ec0677708cb"). InnerVolumeSpecName "kube-api-access-rnmx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:00:25.789807 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:25.788461 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmx9\" (UniqueName: \"kubernetes.io/projected/f905df74-24ab-43d1-9641-7ec0677708cb-kube-api-access-rnmx9\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:25.789807 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:25.788494 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f905df74-24ab-43d1-9641-7ec0677708cb-maas-api-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:00:26.206166 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:00:26.206125 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:00:26.343063 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:26.343027 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" event={"ID":"f905df74-24ab-43d1-9641-7ec0677708cb","Type":"ContainerDied","Data":"e7e9f5da4c288055a83c25df597c80ef336dbeea0de0366076ca9798a8a9d1d6"} Oct 31 15:00:26.343269 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:26.343091 4399 scope.go:122] "RemoveContainer" containerID="65ecd9a0f634785363e55e5eb042124c0f2d334a3adaf69b4503920eac8a9003" Oct 31 15:00:26.343269 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:26.343042 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9" Oct 31 15:00:26.505421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:26.505356 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9"] Oct 31 15:00:26.554758 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:26.554733 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-029d26-29890980-mz9r9"] Oct 31 15:00:26.896416 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:26.896343 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" path="/var/lib/kubelet/pods/f905df74-24ab-43d1-9641-7ec0677708cb/volumes" Oct 31 15:00:33.982285 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.982235 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq"] Oct 31 15:00:33.983018 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.983000 4399 cpu_manager.go:414] "containerMap: removing container" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" containerName="cleanup" Oct 31 15:00:33.983088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.983020 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" containerName="cleanup" Oct 31 15:00:33.983088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.983054 4399 cpu_manager.go:414] "containerMap: removing container" podUID="960dda34-9d5f-4fde-aba0-ca81b683712e" containerName="revoke-keys" Oct 31 15:00:33.983088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.983059 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960dda34-9d5f-4fde-aba0-ca81b683712e" containerName="revoke-keys" Oct 31 15:00:33.983088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.983069 4399 cpu_manager.go:414] "containerMap: removing container" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" containerName="cleanup" Oct 31 15:00:33.983088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.983073 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" containerName="cleanup" Oct 31 15:00:33.983240 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.983145 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" containerName="cleanup" Oct 31 15:00:33.983240 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.983152 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" containerName="cleanup" Oct 31 15:00:33.983240 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.983164 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="960dda34-9d5f-4fde-aba0-ca81b683712e" containerName="revoke-keys" Oct 31 15:00:33.988013 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:33.987998 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.042725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.042681 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq"] Oct 31 15:00:34.076576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.076549 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-plugins-config-volume\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.076746 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.076681 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-routing-overlay-volume\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.076876 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.076851 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-provider-credentials\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.076936 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.076916 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-kube-api-access-hnwzk\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.088993 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.088945 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 31 15:00:34.088993 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.088944 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 31 15:00:34.088993 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.088984 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 31 15:00:34.089183 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.089006 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 31 15:00:34.089183 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.089007 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-79e80514dad4\"" Oct 31 15:00:34.089183 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.089027 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-5e686c13c1d4\"" Oct 31 15:00:34.177783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.177742 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-plugins-config-volume\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.177905 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.177829 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-routing-overlay-volume\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.177905 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.177896 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-provider-credentials\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.177977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.177924 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-kube-api-access-hnwzk\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.178637 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.178555 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-routing-overlay-volume\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.178637 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.178607 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-plugins-config-volume\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.180431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.180408 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-provider-credentials\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.234287 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.234238 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-kube-api-access-hnwzk\") pod \"payload-processing-external-model-7c54d48db8-j6jlq\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.300981 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.300959 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:34.693309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:34.693282 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq"] Oct 31 15:00:34.695471 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:00:34.695444 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351cbc37_0a61_4714_8d7b_17bd381a0c4b.slice/crio-fb11648079e42ae4c3a0bf8bd0a9b3c2b35ff58fb49858acc6f638a26fd8b2e6 WatchSource:0}: Error finding container fb11648079e42ae4c3a0bf8bd0a9b3c2b35ff58fb49858acc6f638a26fd8b2e6: Status 404 returned error can't find the container with id fb11648079e42ae4c3a0bf8bd0a9b3c2b35ff58fb49858acc6f638a26fd8b2e6 Oct 31 15:00:35.407977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:35.407938 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" event={"ID":"351cbc37-0a61-4714-8d7b-17bd381a0c4b","Type":"ContainerStarted","Data":"272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07"} Oct 31 15:00:35.408375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:35.408007 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" event={"ID":"351cbc37-0a61-4714-8d7b-17bd381a0c4b","Type":"ContainerStarted","Data":"fb11648079e42ae4c3a0bf8bd0a9b3c2b35ff58fb49858acc6f638a26fd8b2e6"} Oct 31 15:00:35.408375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:35.408159 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:35.521659 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:35.521611 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" podStartSLOduration=2.521596765 podStartE2EDuration="2.521596765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:35.520556129 +0000 UTC m=+878.731797543" watchObservedRunningTime="2026-10-31 15:00:35.521596765 +0000 UTC m=+878.732838175" Oct 31 15:00:37.786123 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.786080 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg"] Oct 31 15:00:37.786899 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.786879 4399 cpu_manager.go:414] "containerMap: removing container" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" containerName="cleanup" Oct 31 15:00:37.786899 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.786899 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" containerName="cleanup" Oct 31 15:00:37.787072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.787028 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f905df74-24ab-43d1-9641-7ec0677708cb" containerName="cleanup" Oct 31 15:00:37.790591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.790569 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:37.850638 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.850597 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg"] Oct 31 15:00:37.896179 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.896145 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-57815b-openshift-default-dockercfg-9dqr9\"" Oct 31 15:00:37.921371 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.921334 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:37.921567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.921402 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:37.921567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.921537 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:37.921743 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.921604 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:37.921743 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.921647 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8l9q\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-kube-api-access-l8l9q\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:37.921846 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.921804 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:37.921892 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.921875 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:37.921985 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.921969 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:37.922047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:37.922012 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.022682 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.022636 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8l9q\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-kube-api-access-l8l9q\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.022891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.022737 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.022891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.022780 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.022891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.022835 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.022891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.022886 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.023112 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.022940 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.023112 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.023025 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.023112 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.023055 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.023112 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.023103 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.023282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.023171 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.023282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.023231 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.023398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.023312 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.023495 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.023464 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.023884 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.023817 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.025521 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.025496 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.025889 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.025864 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.082023 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.081949 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.082143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.082082 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8l9q\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-kube-api-access-l8l9q\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.105405 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.105374 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:38.538199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.538178 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg"] Oct 31 15:00:38.539662 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:00:38.539632 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e96efc2_aa5d_49b1_87b4_6fb9b01acf37.slice/crio-b64b77615ea80117c5af78e568bfd6876b5cb5da3193fd40e37d18de123da2a3 WatchSource:0}: Error finding container b64b77615ea80117c5af78e568bfd6876b5cb5da3193fd40e37d18de123da2a3: Status 404 returned error can't find the container with id b64b77615ea80117c5af78e568bfd6876b5cb5da3193fd40e37d18de123da2a3 Oct 31 15:00:38.542019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.541988 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:00:38.542086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.542052 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:00:38.542086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:38.542079 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:00:39.431617 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:39.431587 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" event={"ID":"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37","Type":"ContainerStarted","Data":"74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37"} Oct 31 15:00:39.432025 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:39.431651 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" event={"ID":"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37","Type":"ContainerStarted","Data":"b64b77615ea80117c5af78e568bfd6876b5cb5da3193fd40e37d18de123da2a3"} Oct 31 15:00:40.106489 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:40.106457 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:40.108364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:40.108336 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 15:00:40.108507 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:40.108389 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 15:00:41.106061 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:41.106020 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 15:00:41.106533 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:41.106078 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 15:00:42.107171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:42.106404 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 15:00:42.107171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:42.106470 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 15:00:43.071723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.070948 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podStartSLOduration=6.070929537 podStartE2EDuration="6.070929537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:39.657002014 +0000 UTC m=+882.868243426" watchObservedRunningTime="2026-10-31 15:00:43.070929537 +0000 UTC m=+886.282170968" Oct 31 15:00:43.077721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.074235 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp"] Oct 31 15:00:43.098277 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.098247 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.105205 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.105169 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.105370 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.105289 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24x2d\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-kube-api-access-24x2d\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.105370 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.105324 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.105506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.105367 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.105506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.105395 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f2787-9798-4c21-ae4e-7b1136b8b592-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.105506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.105432 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.105506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.105502 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.105751 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.105554 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.105751 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.105584 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.106169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.106144 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 15:00:43.106267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.106194 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 15:00:43.156015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.155986 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2348-openshift-default-dockercfg-k9qcg\"" Oct 31 15:00:43.208374 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.208337 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.209684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.209662 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f2787-9798-4c21-ae4e-7b1136b8b592-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.210571 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.210552 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.210731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.210493 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f2787-9798-4c21-ae4e-7b1136b8b592-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.210850 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.209601 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.211251 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.211231 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.211729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.211691 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.212153 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.212136 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.212632 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.212614 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.213157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.213135 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24x2d\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-kube-api-access-24x2d\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.213628 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.213610 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.214117 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.213001 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.214267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.211615 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.214382 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.212083 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.234309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.234289 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp"] Oct 31 15:00:43.234636 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.234608 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.236977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.236949 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.322308 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.322228 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.337735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.327554 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24x2d\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-kube-api-access-24x2d\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:43.428998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:43.428952 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:44.002066 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:44.001820 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp"] Oct 31 15:00:44.020956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:44.020478 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:00:44.020956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:44.020559 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:00:44.020956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:44.020603 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:00:44.114757 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:44.114689 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 15:00:44.114981 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:44.114949 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 15:00:44.524723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:44.520037 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" event={"ID":"be8f2787-9798-4c21-ae4e-7b1136b8b592","Type":"ContainerStarted","Data":"8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170"} Oct 31 15:00:44.524723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:44.520109 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" event={"ID":"be8f2787-9798-4c21-ae4e-7b1136b8b592","Type":"ContainerStarted","Data":"00350befe1484ac797837fedff498841cc005d5ad3a900ebdc53be66e9d8cc81"} Oct 31 15:00:44.654732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:44.643111 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" podStartSLOduration=1.643090247 podStartE2EDuration="1.643090247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:44.63908195 +0000 UTC m=+887.850323368" watchObservedRunningTime="2026-10-31 15:00:44.643090247 +0000 UTC m=+887.854331660" Oct 31 15:00:45.106322 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:45.106254 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Oct 31 15:00:45.106717 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:45.106583 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Oct 31 15:00:45.430807 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:45.430692 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:45.432427 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:45.432407 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 15:00:45.432487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:45.432453 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 15:00:46.110296 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:46.110269 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:46.110816 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:46.110640 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:46.111423 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:46.111404 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:00:46.417833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:46.417758 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:00:46.430498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:46.430466 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 15:00:46.430620 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:46.430510 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 15:00:47.429646 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:47.429616 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 15:00:47.430141 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:47.429670 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 15:00:48.430196 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:48.430159 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 15:00:48.430734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:48.430222 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 15:00:49.430559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:49.430524 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 15:00:49.430973 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:49.430576 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 15:00:50.430106 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:50.430073 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 31 15:00:50.430290 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:50.430136 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 31 15:00:51.633442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:51.633410 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:51.634208 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:51.634156 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:51.635377 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:51.635357 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:00:55.756896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.756860 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl"] Oct 31 15:00:55.761187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.761164 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.847488 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.847454 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba41a32a-6505-47b2-9fb7-842b885e8369-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.847608 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.847519 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.847608 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.847554 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.847608 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.847575 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.847822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.847749 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.847822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.847794 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.847948 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.847928 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.848009 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.847970 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfs4g\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-kube-api-access-gfs4g\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.848047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.848032 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.893036 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.893005 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl"] Oct 31 15:00:55.949029 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.948999 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949038 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfs4g\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-kube-api-access-gfs4g\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949061 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949123 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba41a32a-6505-47b2-9fb7-842b885e8369-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949335 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949161 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949335 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949196 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949335 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949230 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949489 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949465 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949573 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949630 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949649 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949823 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949761 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.949823 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.949809 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.950169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.950127 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba41a32a-6505-47b2-9fb7-842b885e8369-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.952229 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.952200 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:55.952396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:55.952379 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:56.006171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.006136 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:56.006602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.006579 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfs4g\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-kube-api-access-gfs4g\") pod \"maas-default-gateway-openshift-default-67bf986f99-krbdl\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:56.075336 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.075270 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:56.207334 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:00:56.207298 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:00:56.503763 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.503506 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl"] Oct 31 15:00:56.507498 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:00:56.507471 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba41a32a_6505_47b2_9fb7_842b885e8369.slice/crio-3eb3e2f71e08a7ab5a7d55eae69e04b8aff50d6010e3f9d6edf13385bbe4eec3 WatchSource:0}: Error finding container 3eb3e2f71e08a7ab5a7d55eae69e04b8aff50d6010e3f9d6edf13385bbe4eec3: Status 404 returned error can't find the container with id 3eb3e2f71e08a7ab5a7d55eae69e04b8aff50d6010e3f9d6edf13385bbe4eec3 Oct 31 15:00:56.510185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.510156 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:00:56.510275 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.510218 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:00:56.510275 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.510245 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:00:56.618202 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.618164 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" event={"ID":"ba41a32a-6505-47b2-9fb7-842b885e8369","Type":"ContainerStarted","Data":"c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127"} Oct 31 15:00:56.618384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.618232 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" event={"ID":"ba41a32a-6505-47b2-9fb7-842b885e8369","Type":"ContainerStarted","Data":"3eb3e2f71e08a7ab5a7d55eae69e04b8aff50d6010e3f9d6edf13385bbe4eec3"} Oct 31 15:00:56.765333 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:56.765215 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podStartSLOduration=1.765195445 podStartE2EDuration="1.765195445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:00:56.762105322 +0000 UTC m=+899.973346736" watchObservedRunningTime="2026-10-31 15:00:56.765195445 +0000 UTC m=+899.976436858" Oct 31 15:00:57.076921 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:57.076764 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:00:57.083404 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:57.083378 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 15:00:57.083630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:57.083606 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 15:00:58.077722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:58.077169 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 15:00:58.077722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:58.077292 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 15:00:59.097724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:59.076033 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 15:00:59.097724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:00:59.076091 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 15:01:00.096727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:00.076051 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 15:01:00.096727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:00.076115 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 15:01:01.078065 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:01.078027 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 15:01:01.078473 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:01.078109 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 15:01:02.076617 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:02.076583 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 15:01:02.076824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:02.076646 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 15:01:03.076024 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.075993 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 15:01:03.076442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.076049 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 15:01:03.410569 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.410491 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4"] Oct 31 15:01:03.415434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.415419 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.441228 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.441201 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eff05d-6c38-4c35-a488-5343be8cef30-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.441339 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.441269 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eff05d-6c38-4c35-a488-5343be8cef30-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.441408 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.441388 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-kube-api-access-qrnmd\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.441446 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.441427 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.441491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.441460 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.441602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.441586 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.441672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.441613 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.441672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.441644 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.441804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.441773 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.475330 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.475300 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4"] Oct 31 15:01:03.543122 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543091 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eff05d-6c38-4c35-a488-5343be8cef30-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543297 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543146 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-kube-api-access-qrnmd\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543297 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543188 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543297 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543230 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543295 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543319 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543350 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543658 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543502 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543658 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543556 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eff05d-6c38-4c35-a488-5343be8cef30-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543685 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543750 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543904 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543878 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.543975 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.543944 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.544518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.544481 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eff05d-6c38-4c35-a488-5343be8cef30-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.545644 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.545613 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.545826 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.545805 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eff05d-6c38-4c35-a488-5343be8cef30-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.599828 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.599798 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-kube-api-access-qrnmd\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.600162 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.600146 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:03.729261 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:03.729235 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:04.076671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.076589 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 15:01:04.076671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.076643 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 15:01:04.164750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.164723 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4"] Oct 31 15:01:04.165019 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:01:04.164997 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44eff05d_6c38_4c35_a488_5343be8cef30.slice/crio-d2dc06baa1976aee6e3db2671d2ed32f5d665fa38e427eb6abe0b221e0c11cf1 WatchSource:0}: Error finding container d2dc06baa1976aee6e3db2671d2ed32f5d665fa38e427eb6abe0b221e0c11cf1: Status 404 returned error can't find the container with id d2dc06baa1976aee6e3db2671d2ed32f5d665fa38e427eb6abe0b221e0c11cf1 Oct 31 15:01:04.167358 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.167330 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:01:04.167429 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.167387 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:01:04.167429 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.167416 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:01:04.690510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.690470 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" event={"ID":"44eff05d-6c38-4c35-a488-5343be8cef30","Type":"ContainerStarted","Data":"a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa"} Oct 31 15:01:04.690756 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.690556 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" event={"ID":"44eff05d-6c38-4c35-a488-5343be8cef30","Type":"ContainerStarted","Data":"d2dc06baa1976aee6e3db2671d2ed32f5d665fa38e427eb6abe0b221e0c11cf1"} Oct 31 15:01:04.729550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.729526 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:04.819804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:04.819753 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" podStartSLOduration=1.8197394839999999 podStartE2EDuration="1.819739484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:01:04.818040832 +0000 UTC m=+908.029282254" watchObservedRunningTime="2026-10-31 15:01:04.819739484 +0000 UTC m=+908.030980896" Oct 31 15:01:05.075757 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:05.075719 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 31 15:01:05.075912 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:05.075774 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 31 15:01:05.732136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:05.732101 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 15:01:05.732636 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:05.732167 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 15:01:06.482142 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:06.482103 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:01:06.482730 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:06.482685 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:01:06.483627 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:06.483604 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:01:06.730173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:06.730146 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 15:01:06.730340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:06.730194 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 15:01:07.729548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:07.729520 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 15:01:07.729984 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:07.729567 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 15:01:08.730499 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:08.730463 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 15:01:08.730953 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:08.730545 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 15:01:09.730526 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:09.730494 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 15:01:09.730963 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:09.730548 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 15:01:10.741893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:10.741854 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 31 15:01:10.742515 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:10.742480 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 31 15:01:11.766725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:11.763062 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:11.766725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:11.764494 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:11.766725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:11.765879 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:01:19.762131 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:19.762083 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq"] Oct 31 15:01:24.762392 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:24.762345 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" podUID="351cbc37-0a61-4714-8d7b-17bd381a0c4b" containerName="payload-processing" containerID="cri-o://272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07" gracePeriod=25 Oct 31 15:01:25.033458 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.033436 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:01:25.159854 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.159829 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-plugins-config-volume\") pod \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " Oct 31 15:01:25.160022 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.159880 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-routing-overlay-volume\") pod \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " Oct 31 15:01:25.160022 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.159910 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-kube-api-access-hnwzk\") pod \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " Oct 31 15:01:25.160022 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.160000 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-provider-credentials\") pod \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\" (UID: \"351cbc37-0a61-4714-8d7b-17bd381a0c4b\") " Oct 31 15:01:25.160301 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.160264 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-plugins-config-volume" pod "351cbc37-0a61-4714-8d7b-17bd381a0c4b" (UID: "351cbc37-0a61-4714-8d7b-17bd381a0c4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:01:25.160476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.160287 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-routing-overlay-volume" pod "351cbc37-0a61-4714-8d7b-17bd381a0c4b" (UID: "351cbc37-0a61-4714-8d7b-17bd381a0c4b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:01:25.162387 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.162363 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-provider-credentials" pod "351cbc37-0a61-4714-8d7b-17bd381a0c4b" (UID: "351cbc37-0a61-4714-8d7b-17bd381a0c4b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:01:25.162799 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.162776 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-kube-api-access-hnwzk" pod "351cbc37-0a61-4714-8d7b-17bd381a0c4b" (UID: "351cbc37-0a61-4714-8d7b-17bd381a0c4b"). InnerVolumeSpecName "kube-api-access-hnwzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:01:25.261998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.261967 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:01:25.261998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.261995 4399 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/351cbc37-0a61-4714-8d7b-17bd381a0c4b-routing-overlay-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:01:25.262183 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.262007 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnwzk\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-kube-api-access-hnwzk\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:01:25.262183 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.262015 4399 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/351cbc37-0a61-4714-8d7b-17bd381a0c4b-provider-credentials\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:01:25.880374 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.880337 4399 generic.go:356] "Generic (PLEG): container finished" podID="351cbc37-0a61-4714-8d7b-17bd381a0c4b" containerID="272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07" exitCode=0 Oct 31 15:01:25.880846 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.880386 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" Oct 31 15:01:25.880846 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.880414 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" event={"ID":"351cbc37-0a61-4714-8d7b-17bd381a0c4b","Type":"ContainerDied","Data":"272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07"} Oct 31 15:01:25.880846 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.880466 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq" event={"ID":"351cbc37-0a61-4714-8d7b-17bd381a0c4b","Type":"ContainerDied","Data":"fb11648079e42ae4c3a0bf8bd0a9b3c2b35ff58fb49858acc6f638a26fd8b2e6"} Oct 31 15:01:25.880846 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.880489 4399 scope.go:122] "RemoveContainer" containerID="272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07" Oct 31 15:01:25.892258 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.892065 4399 scope.go:122] "RemoveContainer" containerID="272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07" Oct 31 15:01:25.893809 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:01:25.893780 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07\": container with ID starting with 272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07 not found: ID does not exist" containerID="272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07" Oct 31 15:01:25.893919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:25.893810 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07"} err="failed to get container status \"272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07\": rpc error: code = NotFound desc = could not find container \"272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07\": container with ID starting with 272dce07d108a02ca17870ec198eb8a1dda0c2f91f3d90a7c9cec3e3741d4a07 not found: ID does not exist" Oct 31 15:01:26.043745 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:26.043708 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq"] Oct 31 15:01:26.093064 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:26.093040 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c54d48db8-j6jlq"] Oct 31 15:01:26.206293 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:01:26.206262 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:01:26.898982 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:26.898952 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="351cbc37-0a61-4714-8d7b-17bd381a0c4b" path="/var/lib/kubelet/pods/351cbc37-0a61-4714-8d7b-17bd381a0c4b/volumes" Oct 31 15:01:38.261929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.261852 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8"] Oct 31 15:01:38.262387 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.262373 4399 cpu_manager.go:414] "containerMap: removing container" podUID="351cbc37-0a61-4714-8d7b-17bd381a0c4b" containerName="payload-processing" Oct 31 15:01:38.262433 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.262388 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="351cbc37-0a61-4714-8d7b-17bd381a0c4b" containerName="payload-processing" Oct 31 15:01:38.262511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.262500 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="351cbc37-0a61-4714-8d7b-17bd381a0c4b" containerName="payload-processing" Oct 31 15:01:38.267137 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.267120 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.351530 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.351495 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.351683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.351538 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kj6h\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-kube-api-access-6kj6h\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.351683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.351595 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.351683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.351630 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.351840 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.351689 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.351840 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.351747 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.351840 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.351782 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.351958 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.351839 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.351958 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.351893 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.371773 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.371746 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8"] Oct 31 15:01:38.453051 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453023 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453087 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453134 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453165 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453329 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453208 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kj6h\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-kube-api-access-6kj6h\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453329 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453291 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453443 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453329 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453443 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453400 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453451 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453458 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453587 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453813 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453789 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.453813 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.453804 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.454091 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.454039 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.455766 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.455745 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.455875 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.455861 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.509340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.509308 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.509340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.509336 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kj6h\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-kube-api-access-6kj6h\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:38.580654 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:38.580586 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:39.009748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:39.009721 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8"] Oct 31 15:01:39.011832 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:01:39.011807 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dffa02c_662a_4798_b5c8_e5b5d08ef462.slice/crio-8689160224debf24036bae6022f428b02e6e5b174e4682bb8e4499ac2ed06fb8 WatchSource:0}: Error finding container 8689160224debf24036bae6022f428b02e6e5b174e4682bb8e4499ac2ed06fb8: Status 404 returned error can't find the container with id 8689160224debf24036bae6022f428b02e6e5b174e4682bb8e4499ac2ed06fb8 Oct 31 15:01:39.014096 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:39.014067 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:01:39.014165 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:39.014127 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:01:39.014165 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:39.014158 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:01:39.994980 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:39.994942 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" event={"ID":"7dffa02c-662a-4798-b5c8-e5b5d08ef462","Type":"ContainerStarted","Data":"4a3b781a6e3980a36849982da36642d7c01d8d41cf72333f0723229757780542"} Oct 31 15:01:39.995403 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:39.995032 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" event={"ID":"7dffa02c-662a-4798-b5c8-e5b5d08ef462","Type":"ContainerStarted","Data":"8689160224debf24036bae6022f428b02e6e5b174e4682bb8e4499ac2ed06fb8"} Oct 31 15:01:40.227899 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:40.227846 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podStartSLOduration=2.227830997 podStartE2EDuration="2.227830997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:01:40.175548657 +0000 UTC m=+943.386790286" watchObservedRunningTime="2026-10-31 15:01:40.227830997 +0000 UTC m=+943.439072464" Oct 31 15:01:40.581712 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:40.581662 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:40.583121 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:40.583097 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 15:01:40.583243 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:40.583149 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 15:01:41.581173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:41.581142 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 15:01:41.581559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:41.581206 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 15:01:42.581977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:42.581943 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 15:01:42.582433 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:42.581994 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 15:01:43.581823 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:43.581784 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 15:01:43.582004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:43.581841 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 15:01:44.581797 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:44.581760 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 15:01:44.581978 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:44.581813 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 15:01:45.581568 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:45.581534 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 31 15:01:45.582072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:45.581592 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 31 15:01:46.587312 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:46.587279 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:46.587826 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:46.587720 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:46.588621 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:46.588600 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:01:56.206358 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:01:56.206325 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:01:57.050005 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:01:57.049972 4399 scope.go:122] "RemoveContainer" containerID="997a0605e72d538637e3efdd5f8092ae2afac8eda12951805457e57399e0a38c" Oct 31 15:02:26.209835 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:02:26.209797 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:02:56.206830 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:02:56.206791 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:02:59.128279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.128247 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d"] Oct 31 15:02:59.132293 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.132274 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.237869 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.237836 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d"] Oct 31 15:02:59.267371 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.267334 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.267541 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.267393 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.267541 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.267481 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwmn8\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-kube-api-access-hwmn8\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.267541 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.267516 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.267541 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.267535 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.267685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.267669 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.267759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.267717 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.267759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.267734 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.267838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.267806 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.368665 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.368626 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.368884 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.368688 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.368884 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.368745 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwmn8\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-kube-api-access-hwmn8\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.368884 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.368773 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.368884 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.368797 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.369108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.368939 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.369108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.368985 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.369108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.369014 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.369267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.369159 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.369267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.369184 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.369422 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.369050 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.369673 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.369651 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.369906 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.369676 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.370045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.369739 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.371598 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.371576 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.372103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.372085 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.426036 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.425979 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.426126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.426087 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwmn8\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-kube-api-access-hwmn8\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.445332 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.445308 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:02:59.874606 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.874579 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d"] Oct 31 15:02:59.878750 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:02:59.878500 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d977f9c_25e6_451c_ae90_a0bbc31b2efd.slice/crio-e9b77b61f7d71cd7b5b6a36c186eef3f94dc519fe497241f2f56cdcec14b950d WatchSource:0}: Error finding container e9b77b61f7d71cd7b5b6a36c186eef3f94dc519fe497241f2f56cdcec14b950d: Status 404 returned error can't find the container with id e9b77b61f7d71cd7b5b6a36c186eef3f94dc519fe497241f2f56cdcec14b950d Oct 31 15:02:59.881149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.881113 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:02:59.881255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.881203 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:02:59.881255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:02:59.881248 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:00.461721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:00.460282 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" event={"ID":"2d977f9c-25e6-451c-ae90-a0bbc31b2efd","Type":"ContainerStarted","Data":"973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9"} Oct 31 15:03:00.461721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:00.460361 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" event={"ID":"2d977f9c-25e6-451c-ae90-a0bbc31b2efd","Type":"ContainerStarted","Data":"e9b77b61f7d71cd7b5b6a36c186eef3f94dc519fe497241f2f56cdcec14b950d"} Oct 31 15:03:00.585218 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:00.584477 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" podStartSLOduration=1.584459851 podStartE2EDuration="1.584459851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:02:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:00.582020809 +0000 UTC m=+1023.793262246" watchObservedRunningTime="2026-10-31 15:03:00.584459851 +0000 UTC m=+1023.795701265" Oct 31 15:03:01.448930 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:01.448901 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:03:01.449282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:01.449261 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 15:03:01.449451 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:01.449424 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 15:03:02.460724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:02.453083 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 15:03:02.460724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:02.453145 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 15:03:03.465753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:03.462230 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 15:03:03.465753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:03.462292 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 15:03:04.454725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:04.453094 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 15:03:04.454725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:04.453154 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 15:03:05.450724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:05.450663 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 15:03:05.451307 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:05.451274 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 15:03:06.445962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:06.445932 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body= Oct 31 15:03:06.446134 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:06.445981 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" Oct 31 15:03:07.665197 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:07.664989 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:03:07.665668 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:07.665606 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:03:07.666345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:07.666326 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:03:11.988972 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:11.988940 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c"] Oct 31 15:03:11.993186 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:11.993165 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.044810 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.044780 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-190693\"/\"distinct-w3-190693-kserve-self-signed-certs\"" Oct 31 15:03:12.050412 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.050382 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4vj\" (UniqueName: \"kubernetes.io/projected/14193ed6-7ca5-442b-90d7-b5baff761c16-kube-api-access-5r4vj\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.050531 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.050451 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-home\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.050569 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.050523 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-model-cache\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.050569 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.050561 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-dshm\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.050706 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.050679 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-tmp-dir\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.050746 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.050738 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14193ed6-7ca5-442b-90d7-b5baff761c16-tls-certs\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.097621 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.097593 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c"] Oct 31 15:03:12.152384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.152346 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-home\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.152384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.152387 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-model-cache\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.152630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.152412 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-dshm\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.152630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.152505 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-tmp-dir\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.152630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.152527 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14193ed6-7ca5-442b-90d7-b5baff761c16-tls-certs\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.152802 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.152639 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4vj\" (UniqueName: \"kubernetes.io/projected/14193ed6-7ca5-442b-90d7-b5baff761c16-kube-api-access-5r4vj\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.152860 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.152844 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-home\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.152917 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.152903 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-model-cache\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.152986 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.152962 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-tmp-dir\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.155672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.155647 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14193ed6-7ca5-442b-90d7-b5baff761c16-tls-certs\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.160954 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.160926 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-dshm\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.209378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.209344 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4vj\" (UniqueName: \"kubernetes.io/projected/14193ed6-7ca5-442b-90d7-b5baff761c16-kube-api-access-5r4vj\") pod \"distinct-w3-190693-kserve-6bc94645d6-45x7c\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.314616 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.314526 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:12.709501 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:12.709478 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c"] Oct 31 15:03:12.711398 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:03:12.711372 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14193ed6_7ca5_442b_90d7_b5baff761c16.slice/crio-7e74ab1e41f50a6d36bcaefe83f97c7c9168207e8bca7195d3fd627b43c6288d WatchSource:0}: Error finding container 7e74ab1e41f50a6d36bcaefe83f97c7c9168207e8bca7195d3fd627b43c6288d: Status 404 returned error can't find the container with id 7e74ab1e41f50a6d36bcaefe83f97c7c9168207e8bca7195d3fd627b43c6288d Oct 31 15:03:13.565956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:13.565916 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" event={"ID":"14193ed6-7ca5-442b-90d7-b5baff761c16","Type":"ContainerStarted","Data":"83515f58d50e72661c5e3cd40aa009a4e684fc5897658a89ed64dc758237db02"} Oct 31 15:03:13.565956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:13.565963 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" event={"ID":"14193ed6-7ca5-442b-90d7-b5baff761c16","Type":"ContainerStarted","Data":"7e74ab1e41f50a6d36bcaefe83f97c7c9168207e8bca7195d3fd627b43c6288d"} Oct 31 15:03:13.680242 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:13.680187 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podStartSLOduration=2.680174423 podStartE2EDuration="2.680174423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:13.678845975 +0000 UTC m=+1036.890087400" watchObservedRunningTime="2026-10-31 15:03:13.680174423 +0000 UTC m=+1036.891415835" Oct 31 15:03:14.243182 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.243148 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb"] Oct 31 15:03:14.247220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.247204 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.280624 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.280602 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.280789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.280629 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.280789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.280652 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krggw\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-kube-api-access-krggw\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.280789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.280738 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.281006 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.280878 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.281006 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.280933 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.281078 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.281015 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.281119 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.281075 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.281119 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.281108 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.314906 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.314882 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:14.351450 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.351419 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb"] Oct 31 15:03:14.382736 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.382690 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.382934 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.382763 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.382934 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.382830 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.382934 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.382902 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383071 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383159 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383148 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383209 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383174 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383261 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383208 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383261 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383226 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383261 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383211 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krggw\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-kube-api-access-krggw\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383408 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383315 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383468 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383446 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383526 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383504 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.383908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.383887 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.385846 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.385819 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.385920 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.385885 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.439573 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.439534 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krggw\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-kube-api-access-krggw\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.440782 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.440756 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.561447 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.561370 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:14.988500 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:03:14.988469 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3f9bdf_54b9_44aa_9d6d_8d6d0f2366d1.slice/crio-7b451dbbdc20c467501933f4d2de5df8837e69d8f1a5b01bd52650e737e18e99 WatchSource:0}: Error finding container 7b451dbbdc20c467501933f4d2de5df8837e69d8f1a5b01bd52650e737e18e99: Status 404 returned error can't find the container with id 7b451dbbdc20c467501933f4d2de5df8837e69d8f1a5b01bd52650e737e18e99 Oct 31 15:03:14.990977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.990948 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:14.991048 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.991017 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:14.991085 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.991054 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:14.997683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:14.997658 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb"] Oct 31 15:03:15.580869 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:15.580824 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" event={"ID":"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1","Type":"ContainerStarted","Data":"ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc"} Oct 31 15:03:15.581033 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:15.580894 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" event={"ID":"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1","Type":"ContainerStarted","Data":"7b451dbbdc20c467501933f4d2de5df8837e69d8f1a5b01bd52650e737e18e99"} Oct 31 15:03:15.697906 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:15.697856 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" podStartSLOduration=1.697841048 podStartE2EDuration="1.697841048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:15.695110474 +0000 UTC m=+1038.906351887" watchObservedRunningTime="2026-10-31 15:03:15.697841048 +0000 UTC m=+1038.909082459" Oct 31 15:03:16.566080 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:16.566047 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:16.568077 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:16.568047 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 15:03:16.568223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:16.568103 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 15:03:17.566076 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:17.566040 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 15:03:17.566259 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:17.566110 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 15:03:18.562133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:18.562098 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 15:03:18.562420 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:18.562390 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 15:03:19.566155 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:19.566122 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 15:03:19.566536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:19.566188 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 15:03:20.562475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:20.562037 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 15:03:20.562475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:20.562097 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 15:03:21.562467 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:21.562422 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body= Oct 31 15:03:21.563035 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:21.562482 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" Oct 31 15:03:22.314959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:22.314925 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:22.333400 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:22.333371 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:22.634747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:22.634636 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:03:22.718072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:22.718042 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:22.718745 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:22.718722 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:22.719328 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:22.719311 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:03:26.206477 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:03:26.206441 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:03:34.562483 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.562444 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9"] Oct 31 15:03:34.567442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.567424 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.641433 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.641396 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.641591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.641441 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.641591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.641562 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.641683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.641591 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.641683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.641613 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.641683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.641670 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znww9\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-kube-api-access-znww9\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.641863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.641718 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.641863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.641831 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.641960 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.641931 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.743505 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.743469 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.743742 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.743540 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.743742 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.743655 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.743920 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.743817 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.743920 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.743861 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.744051 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.743952 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.744051 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.743979 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.744051 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.744002 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.744051 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.744029 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znww9\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-kube-api-access-znww9\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.744270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.744122 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.744336 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.744286 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.744432 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.744381 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.744822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.744557 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.744973 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.744882 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.746903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.746884 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.747005 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.746983 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.765441 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.765410 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9"] Oct 31 15:03:34.800202 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.800170 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.800465 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.800440 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znww9\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-kube-api-access-znww9\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:34.882395 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:34.882318 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:35.316618 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:35.316589 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9"] Oct 31 15:03:35.317934 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:03:35.317913 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26232fdc_cf8d_4936_af6d_a8bd6a24e970.slice/crio-4274037ed29d955683fe025f59c152909b8fa19a214f728f8b0247881dcccd35 WatchSource:0}: Error finding container 4274037ed29d955683fe025f59c152909b8fa19a214f728f8b0247881dcccd35: Status 404 returned error can't find the container with id 4274037ed29d955683fe025f59c152909b8fa19a214f728f8b0247881dcccd35 Oct 31 15:03:35.320276 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:35.320242 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:35.320384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:35.320326 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:35.320384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:35.320371 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:35.703750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:35.702956 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" event={"ID":"26232fdc-cf8d-4936-af6d-a8bd6a24e970","Type":"ContainerStarted","Data":"c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4"} Oct 31 15:03:35.703750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:35.703019 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" event={"ID":"26232fdc-cf8d-4936-af6d-a8bd6a24e970","Type":"ContainerStarted","Data":"4274037ed29d955683fe025f59c152909b8fa19a214f728f8b0247881dcccd35"} Oct 31 15:03:35.830042 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:35.829993 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podStartSLOduration=1.829980065 podStartE2EDuration="1.829980065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:35.828649401 +0000 UTC m=+1059.039890816" watchObservedRunningTime="2026-10-31 15:03:35.829980065 +0000 UTC m=+1059.041221477" Oct 31 15:03:35.882851 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:35.882826 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:36.884464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:36.884430 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:03:36.884921 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:36.884480 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 15:03:37.883287 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:37.883256 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:03:37.883467 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:37.883307 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 15:03:38.892721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:38.885109 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:03:38.892721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:38.885182 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 15:03:39.885193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:39.885155 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:03:39.885524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:39.885482 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 15:03:40.885724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:40.885491 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:03:40.885724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:40.885585 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 15:03:41.883008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:41.882978 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:03:41.883190 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:41.883036 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 15:03:42.883029 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:42.882993 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:03:42.883420 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:42.883057 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 15:03:43.909089 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:43.909001 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:43.909661 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:43.909447 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:43.911811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:43.911791 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:03:49.705476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.705440 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw"] Oct 31 15:03:49.711097 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.711075 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.802170 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.802130 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.802314 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.802210 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.802314 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.802281 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.802314 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.802300 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.802451 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.802316 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.802525 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.802499 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.802759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.802737 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.802811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.802781 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hgt\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-kube-api-access-97hgt\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.802847 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.802829 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.815679 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.815651 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw"] Oct 31 15:03:49.903830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.903797 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.903881 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.903898 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hgt\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-kube-api-access-97hgt\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.903927 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.903956 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904237 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.904093 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904237 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.904154 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904237 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.904185 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904237 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.904209 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904458 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.904252 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904458 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.904285 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904572 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.904539 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904628 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.904595 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.904881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.904861 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.906952 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.906928 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.907150 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.907132 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.961383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.961325 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:49.961505 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:49.961398 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hgt\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-kube-api-access-97hgt\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:50.029200 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:50.029173 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:50.460008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:50.459988 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw"] Oct 31 15:03:50.462349 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:03:50.462326 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270f3cc9_fae1_40bd_96aa_0451fd70d10e.slice/crio-400e591f8db6c26d207390bcad7caa14d9157195447aa108158a0beaf2cf6f6d WatchSource:0}: Error finding container 400e591f8db6c26d207390bcad7caa14d9157195447aa108158a0beaf2cf6f6d: Status 404 returned error can't find the container with id 400e591f8db6c26d207390bcad7caa14d9157195447aa108158a0beaf2cf6f6d Oct 31 15:03:50.464894 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:50.464858 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:50.464970 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:50.464920 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:50.464970 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:50.464950 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:50.930384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:50.930288 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" event={"ID":"270f3cc9-fae1-40bd-96aa-0451fd70d10e","Type":"ContainerStarted","Data":"412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432"} Oct 31 15:03:50.930384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:50.930354 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" event={"ID":"270f3cc9-fae1-40bd-96aa-0451fd70d10e","Type":"ContainerStarted","Data":"400e591f8db6c26d207390bcad7caa14d9157195447aa108158a0beaf2cf6f6d"} Oct 31 15:03:51.029958 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:51.029928 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:51.031630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:51.031604 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 15:03:51.031767 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:51.031651 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 15:03:52.030807 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:52.030769 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 15:03:52.031354 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:52.030831 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 15:03:53.029582 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:53.029548 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 15:03:53.029824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:53.029612 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 15:03:54.029522 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.029494 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 15:03:54.029999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.029551 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 15:03:54.295979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.295872 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podStartSLOduration=5.295856799 podStartE2EDuration="5.295856799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:51.066104948 +0000 UTC m=+1074.277346364" watchObservedRunningTime="2026-10-31 15:03:54.295856799 +0000 UTC m=+1077.507098210" Oct 31 15:03:54.297929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.297902 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr"] Oct 31 15:03:54.302087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.302065 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.378306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.378269 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc498\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-kube-api-access-sc498\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.378306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.378309 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.378548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.378334 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.378548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.378450 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.378548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.378518 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ded4067-f432-4509-824e-a862a94f24b3-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.378749 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.378670 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ded4067-f432-4509-824e-a862a94f24b3-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.378749 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.378733 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.378839 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.378797 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.378916 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.378903 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.426172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.426146 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr"] Oct 31 15:03:54.479623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.479590 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ded4067-f432-4509-824e-a862a94f24b3-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.479828 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.479634 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.479828 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.479672 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.479828 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.479804 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.480002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.479849 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc498\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-kube-api-access-sc498\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.480002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.479884 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.480002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.479921 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.480002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.479961 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.480214 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.480038 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ded4067-f432-4509-824e-a862a94f24b3-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.480349 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.480306 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-data\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.480715 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.480653 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-credential-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.480840 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.480818 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ded4067-f432-4509-824e-a862a94f24b3-istiod-ca-cert\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.480966 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.480947 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-socket\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.481254 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.480972 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-certs\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.483731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.483667 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-envoy\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.485653 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.485631 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ded4067-f432-4509-824e-a862a94f24b3-istio-podinfo\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.535995 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.535973 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-istio-token\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.536471 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.536447 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc498\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-kube-api-access-sc498\") pod \"e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:54.616840 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:54.616762 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:55.029857 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:55.029829 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 15:03:55.030248 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:55.029885 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 15:03:55.075753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:55.075720 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr"] Oct 31 15:03:55.076080 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:03:55.076041 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ded4067_f432_4509_824e_a862a94f24b3.slice/crio-56816b21ca196976a6aed032ed15f1f0c7d39121a79446944b4b5d9e964a8ab8 WatchSource:0}: Error finding container 56816b21ca196976a6aed032ed15f1f0c7d39121a79446944b4b5d9e964a8ab8: Status 404 returned error can't find the container with id 56816b21ca196976a6aed032ed15f1f0c7d39121a79446944b4b5d9e964a8ab8 Oct 31 15:03:55.078664 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:55.078637 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:55.078754 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:55.078718 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:55.078754 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:55.078747 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:03:55.999348 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:55.999311 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" event={"ID":"6ded4067-f432-4509-824e-a862a94f24b3","Type":"ContainerStarted","Data":"66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b"} Oct 31 15:03:55.999617 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:55.999371 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" event={"ID":"6ded4067-f432-4509-824e-a862a94f24b3","Type":"ContainerStarted","Data":"56816b21ca196976a6aed032ed15f1f0c7d39121a79446944b4b5d9e964a8ab8"} Oct 31 15:03:56.029755 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:56.029690 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 15:03:56.029905 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:56.029772 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 15:03:56.223959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:56.223901 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" podStartSLOduration=2.223885767 podStartE2EDuration="2.223885767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:03:56.197191584 +0000 UTC m=+1079.408433009" watchObservedRunningTime="2026-10-31 15:03:56.223885767 +0000 UTC m=+1079.435127256" Oct 31 15:03:56.238420 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:03:56.238389 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:03:56.617582 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:56.617550 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:03:56.618751 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:56.618728 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 15:03:56.618882 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:56.618769 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 15:03:57.030235 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:57.030204 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 31 15:03:57.030608 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:57.030256 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 31 15:03:57.617527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:57.617492 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 15:03:57.617731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:57.617541 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 15:03:58.617541 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:58.617508 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 15:03:58.618031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:58.617567 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 15:03:58.641002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:58.640974 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:58.664625 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:58.645741 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:58.666623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:58.666598 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:03:59.608491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.608457 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9"] Oct 31 15:03:59.612855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.612840 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.617395 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.617378 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 15:03:59.617463 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.617419 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 15:03:59.674236 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.674204 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9"] Oct 31 15:03:59.785769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.785732 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8392178-c8ba-4b53-9cec-fe944f3eac47-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.785977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.785800 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.785977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.785884 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.785977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.785939 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.786116 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.786006 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.786116 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.786030 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7g5z\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-kube-api-access-r7g5z\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.786116 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.786075 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.786270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.786192 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.786339 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.786323 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888176 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888085 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888176 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888177 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888224 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8392178-c8ba-4b53-9cec-fe944f3eac47-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888277 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888322 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888371 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888403 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888425 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7g5z\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-kube-api-access-r7g5z\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888795 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888496 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888872 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888845 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888937 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888871 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.888977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.888941 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.889147 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.889128 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.889228 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.889213 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8392178-c8ba-4b53-9cec-fe944f3eac47-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.891436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.891412 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.891535 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.891519 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.966752 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.962865 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:03:59.966752 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:03:59.966491 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7g5z\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-kube-api-access-r7g5z\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:04:00.225874 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:00.225845 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:04:00.617344 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:00.617262 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 15:04:00.617344 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:00.617316 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 15:04:00.693425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:00.693395 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9"] Oct 31 15:04:00.729400 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:04:00.729367 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8392178_c8ba_4b53_9cec_fe944f3eac47.slice/crio-f66e839b5c5b3cfe16ce4a71605927151d432d01a209ce3798cd791df26d7f6e WatchSource:0}: Error finding container f66e839b5c5b3cfe16ce4a71605927151d432d01a209ce3798cd791df26d7f6e: Status 404 returned error can't find the container with id f66e839b5c5b3cfe16ce4a71605927151d432d01a209ce3798cd791df26d7f6e Oct 31 15:04:00.732232 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:00.732204 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:04:00.732309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:00.732286 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:04:00.732355 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:00.732324 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:04:01.033714 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:01.033665 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" event={"ID":"a8392178-c8ba-4b53-9cec-fe944f3eac47","Type":"ContainerStarted","Data":"eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c"} Oct 31 15:04:01.033900 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:01.033738 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" event={"ID":"a8392178-c8ba-4b53-9cec-fe944f3eac47","Type":"ContainerStarted","Data":"f66e839b5c5b3cfe16ce4a71605927151d432d01a209ce3798cd791df26d7f6e"} Oct 31 15:04:01.226566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:01.226538 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:04:01.228198 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:01.228177 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 15:04:01.228303 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:01.228221 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 15:04:01.617463 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:01.617419 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Oct 31 15:04:01.617659 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:01.617483 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Oct 31 15:04:02.226834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:02.226803 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 15:04:02.227182 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:02.226861 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 15:04:03.227110 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:03.227075 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 15:04:03.227538 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:03.227131 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 15:04:03.296153 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:03.296119 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:04:03.296572 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:03.296548 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:04:03.297891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:03.297873 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:04:03.461842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:03.461778 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podStartSLOduration=4.461758698 podStartE2EDuration="4.461758698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:03:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:04:01.162998225 +0000 UTC m=+1084.374239639" watchObservedRunningTime="2026-10-31 15:04:03.461758698 +0000 UTC m=+1086.673000111" Oct 31 15:04:04.235925 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:04.235893 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 15:04:04.236557 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:04.236514 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 15:04:05.226769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:05.226684 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 15:04:05.227088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:05.227054 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 15:04:05.626909 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:05.626800 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:05.783858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:05.783808 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:06.309316 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:06.308858 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 15:04:06.309316 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:06.308922 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 15:04:06.444724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:06.434837 4399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:06.806256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:06.806214 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:04:06.807498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:06.807475 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:04:07.238736 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:07.236629 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 15:04:07.238736 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:07.236719 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 15:04:07.242716 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:07.240230 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:04:08.226672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:08.226631 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body= Oct 31 15:04:08.227099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:08.226723 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" Oct 31 15:04:10.268859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:10.268823 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:04:10.269306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:10.268884 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:10.375345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:10.375315 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:04:10.375953 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:10.375899 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:04:10.377057 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:10.377035 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:04:24.509724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:24.508859 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:24.626723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:24.625788 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:26.208547 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:04:26.208505 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:04:49.634221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:49.632793 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:50.505725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:50.498787 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:50.685846 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:50.685797 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:50.745734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:50.703475 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:04:51.241493 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:51.241448 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:04:51.717994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:51.717876 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:04:52.238942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:04:52.231064 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:04:56.216658 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:04:56.216610 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:05:11.794724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:11.784836 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:11.962663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:11.962628 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt"] Oct 31 15:05:12.111779 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.085917 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.281941 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.262607 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:12.400128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.399774 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt"] Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.455628 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd"] Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.458083 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xbq\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-kube-api-access-j4xbq\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.458142 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.458215 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.458286 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.458342 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.458373 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.458454 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.458576 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.462726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.458641 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdd1426-2ca4-436b-89a4-924805c3f544-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.475589 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.475551 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e8e9c4b-6zcjd"] Oct 31 15:05:12.502843 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.502796 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:12.567559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.567515 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.586864 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.586828 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.587519 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.587495 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.587766 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.587745 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.587930 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.587913 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.588109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.588094 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.588316 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.588299 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.588482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.588466 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdd1426-2ca4-436b-89a4-924805c3f544-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.588670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.588652 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xbq\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-kube-api-access-j4xbq\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.589185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.578824 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.598563 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.591155 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.624378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.592264 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdd1426-2ca4-436b-89a4-924805c3f544-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.624651 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.587388 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.624937 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.592924 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.625077 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.612068 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.710778 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.704755 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xbq\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-kube-api-access-j4xbq\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.728398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.728364 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.750228 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.750162 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lnpkt\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:12.793053 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.791820 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:05:12.793053 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.792176 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:05:12.895726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:12.870996 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:13.029982 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:13.029943 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960dda34-9d5f-4fde-aba0-ca81b683712e" path="/var/lib/kubelet/pods/960dda34-9d5f-4fde-aba0-ca81b683712e/volumes" Oct 31 15:05:13.304537 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:13.300208 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.304537 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:13.301584 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:05:13.319145 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:13.319108 4399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.565141 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:13.565042 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:13.566287 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:13.566264 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:05:13.799761 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:13.798826 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:14.310289 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:14.310244 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:14.742959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:14.689249 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 15:05:15.053089 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:15.049110 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:05:15.244355 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:15.243781 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:15.244355 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:15.243848 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:15.433487 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:05:15.433386 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdd1426_2ca4_436b_89a4_924805c3f544.slice/crio-ef633800c87c8a6904425f76410ac29f5fc4828f9dc87e5d9043f8b9b3882745 WatchSource:0}: Error finding container ef633800c87c8a6904425f76410ac29f5fc4828f9dc87e5d9043f8b9b3882745: Status 404 returned error can't find the container with id ef633800c87c8a6904425f76410ac29f5fc4828f9dc87e5d9043f8b9b3882745 Oct 31 15:05:15.441199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:15.441156 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:05:15.708446 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:15.708395 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:15.778726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:15.726792 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:15.779946 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:15.779908 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:15.780659 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:15.780624 4399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": context deadline exceeded" Oct 31 15:05:15.985868 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:15.977983 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt"] Oct 31 15:05:16.056725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:16.055823 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:16.128723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:16.115341 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:05:16.128723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:16.115509 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:05:16.128723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:16.115592 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:05:16.350340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:16.327675 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:05:16.432901 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:16.432855 4399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:16.464786 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:16.464736 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:17.147730 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:17.143773 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:05:17.346234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:17.346138 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 15:05:17.414777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:17.385992 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:17.414777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:17.386083 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:17.468576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:17.468528 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:17.693858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:17.693821 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:17.694144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:17.694111 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:18.201954 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:05:18.201910 4399 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.267s" Oct 31 15:05:18.202216 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.202186 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" event={"ID":"ccdd1426-2ca4-436b-89a4-924805c3f544","Type":"ContainerStarted","Data":"ef633800c87c8a6904425f76410ac29f5fc4828f9dc87e5d9043f8b9b3882745"} Oct 31 15:05:18.214781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.204602 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:18.364724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.361933 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:18.364724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.362348 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:05:18.472923 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.472812 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:18.820713 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.820603 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" event={"ID":"ccdd1426-2ca4-436b-89a4-924805c3f544","Type":"ContainerStarted","Data":"b9011e43902fe2a6f03a528fa5013d64b9f46d4a9823a1d52b37bb6d23923a99"} Oct 31 15:05:18.911476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.910765 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 15:05:18.911476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.910847 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 15:05:18.951992 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.951831 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:18.951992 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:18.951905 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:19.103102 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:19.102982 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podStartSLOduration=8.102955852 podStartE2EDuration="8.102955852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:19.058888535 +0000 UTC m=+1162.270129949" watchObservedRunningTime="2026-10-31 15:05:19.102955852 +0000 UTC m=+1162.314197265" Oct 31 15:05:19.196949 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:19.196910 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:05:19.197151 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:19.196980 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:05:19.197151 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:19.197004 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:19.516181 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:19.502806 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:05:20.041839 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:20.041806 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 15:05:20.042106 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:20.042075 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 15:05:20.115993 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:20.115961 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:05:20.898580 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:20.898451 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 15:05:20.898580 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:20.898506 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 15:05:21.899017 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:21.898985 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 15:05:21.899393 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:21.899032 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 15:05:22.907047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:22.907005 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 15:05:22.907401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:22.907191 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 15:05:23.898998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:23.898969 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 15:05:23.899174 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:23.899020 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 15:05:24.898959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:24.898928 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 15:05:24.899338 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:24.898975 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 15:05:25.898980 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:25.898952 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 31 15:05:25.899364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:25.898998 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 31 15:05:26.209687 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:05:26.209646 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:05:26.902980 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:26.902956 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:26.903430 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:26.903402 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:26.904019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:26.904003 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:05:35.878397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.878367 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98"] Oct 31 15:05:35.882708 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.882681 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:35.902964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.902937 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4gzb\" (UniqueName: \"kubernetes.io/projected/c9a93516-7ed3-4254-ab04-59dbe7552378-kube-api-access-s4gzb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:35.903083 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.902982 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9a93516-7ed3-4254-ab04-59dbe7552378-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:35.903083 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.903061 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:35.903178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.903158 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:35.903230 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.903214 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:35.903285 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.903274 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a93516-7ed3-4254-ab04-59dbe7552378-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:35.903319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.903301 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:35.937584 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.937517 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 15:05:35.937584 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.937567 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 15:05:35.937975 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.937950 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 15:05:35.938095 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.938042 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jwmrt\"" Oct 31 15:05:35.987429 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:35.987403 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98"] Oct 31 15:05:36.004440 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.004411 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.004575 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.004467 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a93516-7ed3-4254-ab04-59dbe7552378-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.004575 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.004491 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.004736 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.004620 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4gzb\" (UniqueName: \"kubernetes.io/projected/c9a93516-7ed3-4254-ab04-59dbe7552378-kube-api-access-s4gzb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.004736 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.004668 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9a93516-7ed3-4254-ab04-59dbe7552378-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.004830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.004790 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.004888 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.004850 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.005280 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.005259 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.007255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.007083 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.007255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.007219 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c9a93516-7ed3-4254-ab04-59dbe7552378-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.007373 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.007318 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9a93516-7ed3-4254-ab04-59dbe7552378-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.007446 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.007424 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.108722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.104447 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4gzb\" (UniqueName: \"kubernetes.io/projected/c9a93516-7ed3-4254-ab04-59dbe7552378-kube-api-access-s4gzb\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.114009 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.113484 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9a93516-7ed3-4254-ab04-59dbe7552378-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-b2w98\" (UID: \"c9a93516-7ed3-4254-ab04-59dbe7552378\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.202392 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.202361 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:36.715723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.715568 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98"] Oct 31 15:05:36.810109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:36.795607 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 15:05:37.132771 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:37.125785 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" event={"ID":"c9a93516-7ed3-4254-ab04-59dbe7552378","Type":"ContainerStarted","Data":"f360a341e3aa35b145f9ea7e1dc794820e81cc5073c3a6f49ebf321887edd5c7"} Oct 31 15:05:38.168735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:38.167989 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:38.516725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:38.501486 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:05:38.805895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:38.805792 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:38.823725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:38.821813 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:38.823725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:38.821891 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:39.170776 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:39.170644 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:39.172134 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:39.172109 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:05:39.509842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:39.509794 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:05:39.510883 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:39.510857 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:05:39.838794 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:39.831818 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:05:39.838794 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:39.833434 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:05:40.181911 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:40.181805 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:40.498455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:40.498421 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:05:40.927775 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:40.839939 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:05:41.188780 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:41.185841 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:41.555215 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:41.555169 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:05:41.897188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:41.896778 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:41.897188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:41.896843 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:41.930482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:41.928792 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.212781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:42.195817 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.567777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:42.556387 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:42.567777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:42.556813 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:05:42.930879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:42.929709 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:05:42.946863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:42.946805 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:05:42.947052 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:42.946870 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:43.198778 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:43.196015 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:43.345723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:43.345652 4399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:05:43.392549 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:43.391373 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:05:43.438522 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:43.437230 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:05:43.636143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:43.636064 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:05:45.878565 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:45.878537 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:05:45.878874 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:45.878612 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:05:46.744585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:46.744550 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" event={"ID":"c9a93516-7ed3-4254-ab04-59dbe7552378","Type":"ContainerStarted","Data":"7c07ae53172dc52c5aefb6e9bae388fbc94df7eccc7ef151d4440c8702faa93c"} Oct 31 15:05:46.745033 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:46.745006 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:46.746995 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:46.746968 4399 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-b2w98 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:05:46.747133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:46.747018 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" podUID="c9a93516-7ed3-4254-ab04-59dbe7552378" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:05:46.944493 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:46.944439 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" podStartSLOduration=2.862961403 podStartE2EDuration="11.944425129s" totalImagesPullingTime="9.081463726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:05:46.904765982 +0000 UTC m=+1190.116007393" watchObservedRunningTime="2026-10-31 15:05:46.944425129 +0000 UTC m=+1190.155666541" Oct 31 15:05:47.750789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:47.750761 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-b2w98" Oct 31 15:05:56.231015 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:05:56.230973 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:05:59.945725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:05:59.944942 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.439783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:00.438783 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:00.963724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:00.959838 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:01.039474 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:01.039435 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:06:01.517725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:01.517207 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:01.631721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:01.571736 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:06:01.989731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:01.982531 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.199211 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:02.198595 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:06:02.199211 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:02.198684 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.211279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:02.211236 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:02.517967 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:02.517794 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:06:02.641473 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:02.628480 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:06:02.649111 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:02.649050 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:06:26.208852 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:06:26.208817 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:06:37.528889 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:37.528852 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-krbdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:06:37.529325 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:37.528911 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:06:56.208807 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:06:56.208758 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:06:57.118390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:06:57.118362 4399 scope.go:122] "RemoveContainer" containerID="b374ca6b5bd6b32a97e0c8efe34216cb891a09be0b8cd53ba6516f84838bfab1" Oct 31 15:07:00.097726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:00.037180 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:00.658860 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:00.658808 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:07:01.132725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:01.120918 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:01.162804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:01.141111 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:07:01.217843 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:01.217780 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 15:07:01.682851 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:01.682802 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:07:01.697178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:01.697145 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:07:02.233481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:02.233442 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.290995 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:02.289783 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:02.296738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:02.291793 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:07:02.693630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:02.693528 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.290868 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:03.290823 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.320883 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:03.320778 4399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.342728 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:03.335245 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:03.720548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:03.720507 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded" Oct 31 15:07:03.881677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:03.881231 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:07:03.881677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:03.881294 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:07:04.233888 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:04.233677 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:07:04.318318 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:04.318276 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:07:04.720799 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:04.720419 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:07:04.820339 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:04.820306 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:07:08.674835 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.674805 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6"] Oct 31 15:07:08.680401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.680379 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.737223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.737197 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6"] Oct 31 15:07:08.774249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.774211 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.774418 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.774271 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.774460 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.774414 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.774508 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.774463 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.774546 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.774533 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ngjs\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-kube-api-access-2ngjs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.774587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.774567 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.774731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.774678 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.774825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.774806 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.774915 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.774902 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.875919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.875886 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.875919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.875925 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.875973 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876001 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876018 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ngjs\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-kube-api-access-2ngjs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876040 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876104 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876171 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876310 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876337 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876455 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876536 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876608 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.876744 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.876728 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.878644 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.878619 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.878765 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.878748 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.932793 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.932727 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.932918 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.932812 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ngjs\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-kube-api-access-2ngjs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:08.994166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:08.994137 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:09.441902 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:09.441873 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6"] Oct 31 15:07:09.446083 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:09.446052 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:07:09.446219 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:09.446130 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:07:09.446219 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:09.446175 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:07:09.726076 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:09.726038 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" event={"ID":"7cae3859-4f99-4a41-88f9-fc6ae17e301c","Type":"ContainerStarted","Data":"8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367"} Oct 31 15:07:09.726516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:09.726099 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" event={"ID":"7cae3859-4f99-4a41-88f9-fc6ae17e301c","Type":"ContainerStarted","Data":"47f2ca91f67ffee51d9d282d705eb9f8b9d104f75b1349dcdcb4c1f3d1d2a765"} Oct 31 15:07:09.845931 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:09.845880 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" podStartSLOduration=1.845864513 podStartE2EDuration="1.845864513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:09.843662874 +0000 UTC m=+1273.054904287" watchObservedRunningTime="2026-10-31 15:07:09.845864513 +0000 UTC m=+1273.057105924" Oct 31 15:07:09.995175 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:09.995094 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:10.995880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:10.995851 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:07:10.996350 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:10.995897 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 15:07:11.994536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:11.994501 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:07:11.994735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:11.994551 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 15:07:12.994415 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:12.994382 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:07:12.994827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:12.994436 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 15:07:13.995267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:13.995234 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:07:13.995791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:13.995293 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 15:07:14.994944 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:14.994910 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:07:14.995105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:14.994962 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 15:07:15.994672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:15.994640 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Oct 31 15:07:15.995091 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:15.994710 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Oct 31 15:07:17.000019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:16.999990 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:17.000453 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:17.000428 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:17.001415 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:17.001387 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:07:26.206853 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:07:26.206818 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:07:38.954610 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:38.954540 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82"] Oct 31 15:07:38.958645 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:38.958623 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.029518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.029487 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.029668 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.029526 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.029668 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.029602 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.029668 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.029640 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8461d1f-5523-47e3-a74f-9ceae9650c08-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.029800 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.029712 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.029800 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.029758 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8gn2\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-kube-api-access-s8gn2\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.029897 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.029876 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.029935 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.029917 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.029976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.029939 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131204 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131173 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131218 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131233 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131313 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131344 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131589 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131405 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131589 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131466 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8461d1f-5523-47e3-a74f-9ceae9650c08-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131589 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131515 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131794 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131587 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8gn2\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-kube-api-access-s8gn2\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131794 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131614 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131794 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131785 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131948 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131832 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.131948 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.131870 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.132302 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.132278 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8461d1f-5523-47e3-a74f-9ceae9650c08-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.133921 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.133900 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.134067 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.134048 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.164261 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.164232 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82"] Oct 31 15:07:39.188591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.188558 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.188743 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.188662 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8gn2\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-kube-api-access-s8gn2\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.275147 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.275075 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:39.710407 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.710380 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82"] Oct 31 15:07:39.711725 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:07:39.711675 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8461d1f_5523_47e3_a74f_9ceae9650c08.slice/crio-05bd0745ebce25c7d9408620653fb03c3eab15b48a3f266932e57922697dbccd WatchSource:0}: Error finding container 05bd0745ebce25c7d9408620653fb03c3eab15b48a3f266932e57922697dbccd: Status 404 returned error can't find the container with id 05bd0745ebce25c7d9408620653fb03c3eab15b48a3f266932e57922697dbccd Oct 31 15:07:39.714170 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.714136 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:07:39.714262 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.714201 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:07:39.714262 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:39.714229 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:07:40.121604 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:40.121506 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" event={"ID":"a8461d1f-5523-47e3-a74f-9ceae9650c08","Type":"ContainerStarted","Data":"0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6"} Oct 31 15:07:40.121604 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:40.121565 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" event={"ID":"a8461d1f-5523-47e3-a74f-9ceae9650c08","Type":"ContainerStarted","Data":"05bd0745ebce25c7d9408620653fb03c3eab15b48a3f266932e57922697dbccd"} Oct 31 15:07:40.276341 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:40.276302 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:40.277721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:40.277681 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 15:07:40.277867 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:40.277744 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 15:07:40.291101 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:40.291051 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podStartSLOduration=2.291034706 podStartE2EDuration="2.291034706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:07:40.290767282 +0000 UTC m=+1303.502008693" watchObservedRunningTime="2026-10-31 15:07:40.291034706 +0000 UTC m=+1303.502276121" Oct 31 15:07:41.275690 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:41.275656 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 15:07:41.276108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:41.275739 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 15:07:42.276206 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:42.276169 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 15:07:42.276762 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:42.276228 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 15:07:43.275975 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:43.275941 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 15:07:43.276160 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:43.275990 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 15:07:44.276435 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:44.276402 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 15:07:44.276854 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:44.276468 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 15:07:45.275653 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:45.275618 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 15:07:45.275858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:45.275685 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 15:07:46.275519 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:46.275480 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Oct 31 15:07:46.276036 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:46.275552 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Oct 31 15:07:47.638772 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:47.638740 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:47.639852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:47.639824 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:47.641941 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:07:47.641907 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:07:56.207689 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:07:56.207654 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:08:26.207126 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:08:26.207086 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:08:26.733083 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.733041 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8"] Oct 31 15:08:26.737234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.737202 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:26.801741 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.801498 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8"] Oct 31 15:08:26.847006 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.846967 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4d3ea8-dockercfg-fsflq\"" Oct 31 15:08:26.847209 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.847073 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4d3ea8\"" Oct 31 15:08:26.847706 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.847663 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxt8c\" (UniqueName: \"kubernetes.io/projected/dc557f5b-3084-49e7-baf6-18fb698ca7cd-kube-api-access-sxt8c\") pod \"payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8\" (UID: \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:26.848336 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.848012 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc557f5b-3084-49e7-baf6-18fb698ca7cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8\" (UID: \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:26.903076 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.903046 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd"] Oct 31 15:08:26.906804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.906767 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:26.952815 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.952768 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc557f5b-3084-49e7-baf6-18fb698ca7cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8\" (UID: \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:26.954119 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.954086 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwsrq\" (UniqueName: \"kubernetes.io/projected/a4c6f555-76aa-41ee-a5f1-1829292fe076-kube-api-access-nwsrq\") pod \"payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd\" (UID: \"a4c6f555-76aa-41ee-a5f1-1829292fe076\") " pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:26.954383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.954362 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxt8c\" (UniqueName: \"kubernetes.io/projected/dc557f5b-3084-49e7-baf6-18fb698ca7cd-kube-api-access-sxt8c\") pod \"payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8\" (UID: \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:26.954535 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.954510 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c6f555-76aa-41ee-a5f1-1829292fe076-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd\" (UID: \"a4c6f555-76aa-41ee-a5f1-1829292fe076\") " pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:26.954902 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.953933 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc557f5b-3084-49e7-baf6-18fb698ca7cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8\" (UID: \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:26.959200 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:26.959180 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd"] Oct 31 15:08:27.056497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:27.056404 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c6f555-76aa-41ee-a5f1-1829292fe076-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd\" (UID: \"a4c6f555-76aa-41ee-a5f1-1829292fe076\") " pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:27.057493 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:27.057446 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwsrq\" (UniqueName: \"kubernetes.io/projected/a4c6f555-76aa-41ee-a5f1-1829292fe076-kube-api-access-nwsrq\") pod \"payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd\" (UID: \"a4c6f555-76aa-41ee-a5f1-1829292fe076\") " pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:27.058255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:27.058223 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c6f555-76aa-41ee-a5f1-1829292fe076-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd\" (UID: \"a4c6f555-76aa-41ee-a5f1-1829292fe076\") " pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:27.063846 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:27.063663 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxt8c\" (UniqueName: \"kubernetes.io/projected/dc557f5b-3084-49e7-baf6-18fb698ca7cd-kube-api-access-sxt8c\") pod \"payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8\" (UID: \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:27.114498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:27.114463 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwsrq\" (UniqueName: \"kubernetes.io/projected/a4c6f555-76aa-41ee-a5f1-1829292fe076-kube-api-access-nwsrq\") pod \"payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd\" (UID: \"a4c6f555-76aa-41ee-a5f1-1829292fe076\") " pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:27.222058 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:27.222019 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:27.354063 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:27.353163 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:27.638819 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:27.638783 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd"] Oct 31 15:08:27.787045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:27.777326 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8"] Oct 31 15:08:28.557726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:28.557129 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" event={"ID":"a4c6f555-76aa-41ee-a5f1-1829292fe076","Type":"ContainerStarted","Data":"e310ca93f9547fc49d5645c83680c9ac301aa3ec32204f109dbc0ef75ded8431"} Oct 31 15:08:28.557726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:28.557203 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" event={"ID":"a4c6f555-76aa-41ee-a5f1-1829292fe076","Type":"ContainerStarted","Data":"b97765453ca58f764e8fadb8809f5e7b84a50e9825ad5f328eedfb14f03c8345"} Oct 31 15:08:28.582725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:28.570404 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:28.582725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:28.573025 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" event={"ID":"dc557f5b-3084-49e7-baf6-18fb698ca7cd","Type":"ContainerStarted","Data":"c47ee95299cbbbf318df4461c09f148d6571ae5abd56ca061af1dc386fa529a1"} Oct 31 15:08:28.582725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:28.573079 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" event={"ID":"dc557f5b-3084-49e7-baf6-18fb698ca7cd","Type":"ContainerStarted","Data":"8c5b213d3164b36a1b142c14d18a6d0ba74feaa48547d11237102157195e51ec"} Oct 31 15:08:28.582725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:28.573885 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:28.689735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:28.688938 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" podStartSLOduration=2.688919164 podStartE2EDuration="2.688919164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:28.687090968 +0000 UTC m=+1351.898332382" watchObservedRunningTime="2026-10-31 15:08:28.688919164 +0000 UTC m=+1351.900160577" Oct 31 15:08:29.744777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:29.744014 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:29.824832 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:29.824784 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:30.341724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:30.322066 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:08:30.754566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:30.752820 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:30.754566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:30.753369 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:08:31.043805 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:31.040433 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" podStartSLOduration=5.040412569 podStartE2EDuration="5.040412569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:28.815060032 +0000 UTC m=+1352.026301446" watchObservedRunningTime="2026-10-31 15:08:31.040412569 +0000 UTC m=+1354.251653983" Oct 31 15:08:31.352783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:31.329411 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:31.355243 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:31.353394 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:08:31.809735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:31.783830 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:32.474723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.473352 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:08:32.511999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.511958 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r"] Oct 31 15:08:32.644987 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.644940 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r"] Oct 31 15:08:32.645241 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.645211 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.645401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.645385 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:08:32.645559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.645548 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:08:32.748387 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.748288 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cf5d51-openshift-default-dockercfg-rrffc\"" Oct 31 15:08:32.846422 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.846392 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-certs\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.846881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.846468 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-envoy\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.846881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.846514 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-socket\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.846881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.846782 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcf99\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-kube-api-access-lcf99\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.847015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.846977 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-credential-socket\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.847058 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.847016 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-token\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.847148 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.847125 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-podinfo\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.847193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.847176 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istiod-ca-cert\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.847275 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.847260 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-data\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.949727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.949634 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-data\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.950479 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.950447 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-certs\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.950611 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.950524 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-certs\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.950863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.950841 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-envoy\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.951396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.951356 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-socket\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.951498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.951473 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-data\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.951561 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.951534 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcf99\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-kube-api-access-lcf99\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.951802 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.951681 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-credential-socket\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.951802 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.951729 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-token\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.951802 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.951788 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-podinfo\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.951998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.951825 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istiod-ca-cert\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.951998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.951847 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-socket\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.952481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.952456 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-credential-socket\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.953301 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.953276 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istiod-ca-cert\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.956747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.956683 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-envoy\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:32.958449 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:32.957385 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-podinfo\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:33.013613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:33.013504 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-token\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:33.016838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:33.016780 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcf99\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-kube-api-access-lcf99\") pod \"e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:33.298103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:33.298015 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:33.774744 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:33.774715 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r"] Oct 31 15:08:33.775587 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:08:33.775563 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6494ac2c_f5da_4c61_8d14_b97acfe5aedf.slice/crio-4d5705b76046f71180f84886e9760b1939fa71e8e9c4ce419d79e5c040766f10 WatchSource:0}: Error finding container 4d5705b76046f71180f84886e9760b1939fa71e8e9c4ce419d79e5c040766f10: Status 404 returned error can't find the container with id 4d5705b76046f71180f84886e9760b1939fa71e8e9c4ce419d79e5c040766f10 Oct 31 15:08:33.777943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:33.777913 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:08:33.778015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:33.777980 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:08:33.778015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:33.778007 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:08:34.193942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:34.193861 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" event={"ID":"6494ac2c-f5da-4c61-8d14-b97acfe5aedf","Type":"ContainerStarted","Data":"0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39"} Oct 31 15:08:34.193942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:34.193925 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" event={"ID":"6494ac2c-f5da-4c61-8d14-b97acfe5aedf","Type":"ContainerStarted","Data":"4d5705b76046f71180f84886e9760b1939fa71e8e9c4ce419d79e5c040766f10"} Oct 31 15:08:34.299085 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:34.299049 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:34.304728 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:34.304678 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:34.312991 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:34.312940 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" podStartSLOduration=3.31292729 podStartE2EDuration="3.31292729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:34.308788094 +0000 UTC m=+1357.520029516" watchObservedRunningTime="2026-10-31 15:08:34.31292729 +0000 UTC m=+1357.524168701" Oct 31 15:08:35.204439 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:35.204403 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:35.205436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:35.205418 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:08:40.837104 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:40.837072 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:08:40.844989 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:40.844963 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:08:40.943995 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:40.943969 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr"] Oct 31 15:08:40.948610 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:40.948590 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:40.995154 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:40.995110 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr"] Oct 31 15:08:41.002991 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.002964 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cf5d51-dockercfg-wz4hn\"" Oct 31 15:08:41.003914 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.003319 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cf5d51\"" Oct 31 15:08:41.110215 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.110102 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld74f\" (UniqueName: \"kubernetes.io/projected/b15498a0-5e44-465f-af26-41b5130d5bfc-kube-api-access-ld74f\") pod \"payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr\" (UID: \"b15498a0-5e44-465f-af26-41b5130d5bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:41.110490 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.110305 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15498a0-5e44-465f-af26-41b5130d5bfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr\" (UID: \"b15498a0-5e44-465f-af26-41b5130d5bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:41.212520 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.212485 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15498a0-5e44-465f-af26-41b5130d5bfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr\" (UID: \"b15498a0-5e44-465f-af26-41b5130d5bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:41.213030 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.213003 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld74f\" (UniqueName: \"kubernetes.io/projected/b15498a0-5e44-465f-af26-41b5130d5bfc-kube-api-access-ld74f\") pod \"payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr\" (UID: \"b15498a0-5e44-465f-af26-41b5130d5bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:41.213454 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.213429 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15498a0-5e44-465f-af26-41b5130d5bfc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr\" (UID: \"b15498a0-5e44-465f-af26-41b5130d5bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:41.274455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.274418 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld74f\" (UniqueName: \"kubernetes.io/projected/b15498a0-5e44-465f-af26-41b5130d5bfc-kube-api-access-ld74f\") pod \"payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr\" (UID: \"b15498a0-5e44-465f-af26-41b5130d5bfc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:41.569511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.569474 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:41.960380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:41.960352 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr"] Oct 31 15:08:41.962593 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:08:41.962567 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb15498a0_5e44_465f_af26_41b5130d5bfc.slice/crio-062a7c9454268e92cf1759e58cca7f60743822fdaa176e7a66aa117b320bb67f WatchSource:0}: Error finding container 062a7c9454268e92cf1759e58cca7f60743822fdaa176e7a66aa117b320bb67f: Status 404 returned error can't find the container with id 062a7c9454268e92cf1759e58cca7f60743822fdaa176e7a66aa117b320bb67f Oct 31 15:08:42.289689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:42.289585 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" event={"ID":"b15498a0-5e44-465f-af26-41b5130d5bfc","Type":"ContainerStarted","Data":"1709f464086e4102b65ce63cf195525c47407c6d9bc84fd85e914c647f508f79"} Oct 31 15:08:42.289689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:42.289683 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" event={"ID":"b15498a0-5e44-465f-af26-41b5130d5bfc","Type":"ContainerStarted","Data":"062a7c9454268e92cf1759e58cca7f60743822fdaa176e7a66aa117b320bb67f"} Oct 31 15:08:42.289936 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:42.289820 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:42.402792 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:42.402743 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" podStartSLOduration=2.402729582 podStartE2EDuration="2.402729582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:42.401939384 +0000 UTC m=+1365.613180807" watchObservedRunningTime="2026-10-31 15:08:42.402729582 +0000 UTC m=+1365.613970994" Oct 31 15:08:47.715029 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.714997 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7"] Oct 31 15:08:47.719756 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.719734 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.776294 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.776273 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5695787b-openshift-default-dockercfg-zpbf4\"" Oct 31 15:08:47.801511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.801483 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.801664 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.801543 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.801664 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.801630 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brgnx\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-kube-api-access-brgnx\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.801664 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.801660 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.801842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.801733 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.801883 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.801848 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25234fe-9717-4f3c-874f-cac85c87cf6c-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.801930 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.801916 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.802040 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.802023 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.802087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.802054 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.826482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.826455 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7"] Oct 31 15:08:47.903682 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.903647 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brgnx\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-kube-api-access-brgnx\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.903682 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.903691 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.903950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.903746 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.903950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.903778 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25234fe-9717-4f3c-874f-cac85c87cf6c-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.903950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.903917 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.904119 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.904016 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.904119 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.904042 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.904225 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.904159 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.904225 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.904164 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.904318 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.904239 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.904318 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.904268 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.904505 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.904482 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.904576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.904520 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.904576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.904547 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25234fe-9717-4f3c-874f-cac85c87cf6c-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.906919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.906892 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.907169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.907150 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.960150 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.960126 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:47.960257 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:47.960162 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brgnx\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-kube-api-access-brgnx\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:48.035560 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:48.035495 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:48.475362 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:48.475337 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7"] Oct 31 15:08:48.476570 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:08:48.476543 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda25234fe_9717_4f3c_874f_cac85c87cf6c.slice/crio-5ce65a9bc5e42c80332a2cc753835d5cdf2808310b0d52406c2aa2cd8ba690d3 WatchSource:0}: Error finding container 5ce65a9bc5e42c80332a2cc753835d5cdf2808310b0d52406c2aa2cd8ba690d3: Status 404 returned error can't find the container with id 5ce65a9bc5e42c80332a2cc753835d5cdf2808310b0d52406c2aa2cd8ba690d3 Oct 31 15:08:48.479354 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:48.479320 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:08:48.479434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:48.479390 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:08:48.479434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:48.479427 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:08:49.339613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:49.339577 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" event={"ID":"a25234fe-9717-4f3c-874f-cac85c87cf6c","Type":"ContainerStarted","Data":"1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121"} Oct 31 15:08:49.340027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:49.339635 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" event={"ID":"a25234fe-9717-4f3c-874f-cac85c87cf6c","Type":"ContainerStarted","Data":"5ce65a9bc5e42c80332a2cc753835d5cdf2808310b0d52406c2aa2cd8ba690d3"} Oct 31 15:08:49.562155 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:49.562088 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" podStartSLOduration=2.562067803 podStartE2EDuration="2.562067803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:08:49.505890999 +0000 UTC m=+1372.717132449" watchObservedRunningTime="2026-10-31 15:08:49.562067803 +0000 UTC m=+1372.773309252" Oct 31 15:08:50.035941 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:50.035904 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:50.037309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:50.037287 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 15:08:50.037387 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:50.037336 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 15:08:51.035943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:51.035911 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 15:08:51.036417 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:51.035965 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 15:08:52.036449 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:52.036417 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 15:08:52.036884 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:52.036472 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 15:08:53.035848 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:53.035808 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 15:08:53.036032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:53.035869 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 15:08:53.301198 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:53.301129 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:08:54.036168 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:54.036134 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 15:08:54.036350 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:54.036210 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 15:08:55.036315 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:55.036278 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Oct 31 15:08:55.036739 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:55.036356 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Oct 31 15:08:56.039786 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:56.039754 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:56.040133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:56.040013 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:56.040620 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:08:56.040599 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:08:56.212336 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:08:56.212296 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:09:20.458143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.458111 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk"] Oct 31 15:09:20.462583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.462566 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.527417 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.527394 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-metrics-certs\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.527571 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.527520 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.527657 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.527640 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xz79\" (UniqueName: \"kubernetes.io/projected/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-kube-api-access-9xz79\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.527737 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.527721 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-openshift-service-ca\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.563950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.563926 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cf5d51-serving-cert\"" Oct 31 15:09:20.567536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.567515 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk"] Oct 31 15:09:20.629188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.629160 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xz79\" (UniqueName: \"kubernetes.io/projected/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-kube-api-access-9xz79\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.629353 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.629258 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-openshift-service-ca\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.629353 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.629347 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-metrics-certs\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.629490 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.629476 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.629622 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:09:20.629600 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-serving-cert: secret "maas-api-e2e-api-b-cf5d51-serving-cert" not found Oct 31 15:09:20.629741 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:09:20.629726 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls podName:e1ea3563-2e5b-492b-bcd2-1b6caaf997f0 nodeName:}" failed. No retries permitted until 2026-10-31 15:09:21.129691259 +0000 UTC m=+1404.340932665 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls") pod "maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" (UID: "e1ea3563-2e5b-492b-bcd2-1b6caaf997f0") : secret "maas-api-e2e-api-b-cf5d51-serving-cert" not found Oct 31 15:09:20.629905 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.629887 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-openshift-service-ca\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.631908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.631880 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-metrics-certs\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:20.688411 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:20.688387 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xz79\" (UniqueName: \"kubernetes.io/projected/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-kube-api-access-9xz79\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:21.137119 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:21.137087 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:21.139648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:21.139624 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls\") pod \"maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:21.376781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:21.376753 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:21.755279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:21.755252 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk"] Oct 31 15:09:21.756714 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:09:21.756672 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ea3563_2e5b_492b_bcd2_1b6caaf997f0.slice/crio-82ff7a03b607144bdd1d9e192648ce7a26b69e11bd3944a157f1abdc1c0a1968 WatchSource:0}: Error finding container 82ff7a03b607144bdd1d9e192648ce7a26b69e11bd3944a157f1abdc1c0a1968: Status 404 returned error can't find the container with id 82ff7a03b607144bdd1d9e192648ce7a26b69e11bd3944a157f1abdc1c0a1968 Oct 31 15:09:22.617605 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:22.617537 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" event={"ID":"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0","Type":"ContainerStarted","Data":"82ff7a03b607144bdd1d9e192648ce7a26b69e11bd3944a157f1abdc1c0a1968"} Oct 31 15:09:23.673722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:23.666091 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" event={"ID":"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0","Type":"ContainerStarted","Data":"bc2c6d46bfb1aa470313acdfc3c878586365ebbaad204d1def354bedc5d0206d"} Oct 31 15:09:23.673722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:23.670619 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:23.832719 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:23.824919 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" podStartSLOduration=3.045653648 podStartE2EDuration="3.824880595s" totalImagesPullingTime="779.226947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:09:23.820147125 +0000 UTC m=+1407.031388538" watchObservedRunningTime="2026-10-31 15:09:23.824880595 +0000 UTC m=+1407.036122014" Oct 31 15:09:26.319945 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:09:26.319873 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:09:30.743048 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:09:30.743022 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:09:56.229354 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:09:56.229319 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:10:12.072291 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:12.072260 4399 generic.go:356] "Generic (PLEG): container finished" podID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerID="0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39" exitCode=0 Oct 31 15:10:12.072665 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:12.072336 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" event={"ID":"6494ac2c-f5da-4c61-8d14-b97acfe5aedf","Type":"ContainerDied","Data":"0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39"} Oct 31 15:10:12.072901 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:12.072885 4399 scope.go:122] "RemoveContainer" containerID="0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39" Oct 31 15:10:12.074683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:12.074653 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:10:12.074812 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:12.074793 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:10:12.074856 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:12.074845 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:10:13.081951 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:13.081915 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" event={"ID":"6494ac2c-f5da-4c61-8d14-b97acfe5aedf","Type":"ContainerStarted","Data":"d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6"} Oct 31 15:10:13.299148 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:13.299121 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:10:13.299315 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:13.299265 4399 patch_prober.go:29] interesting pod/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:10:13.299360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:13.299308 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 15:10:14.298504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:14.298470 4399 patch_prober.go:29] interesting pod/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:10:14.299017 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:14.298532 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 15:10:15.300868 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:15.300350 4399 patch_prober.go:29] interesting pod/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:10:15.300868 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:15.300411 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 15:10:16.299079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:16.299039 4399 patch_prober.go:29] interesting pod/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:10:16.299417 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:16.299388 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 15:10:17.303886 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:17.303846 4399 patch_prober.go:29] interesting pod/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 31 15:10:17.304507 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:17.304472 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 31 15:10:18.568971 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:18.568938 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:10:18.569925 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:18.569871 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:10:18.571020 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:18.570995 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:10:21.330836 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.330801 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp"] Oct 31 15:10:21.335648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.335624 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.390500 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.390468 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-644a8a-openshift-default-dockercfg-5qxv9\"" Oct 31 15:10:21.408576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.408535 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb2gj\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-kube-api-access-zb2gj\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.409537 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.409510 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.409680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.409555 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.409680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.409638 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.409848 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.409690 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.409848 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.409778 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.409848 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.409841 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.410004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.409892 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.410004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.409925 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.486487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.486454 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp"] Oct 31 15:10:21.511899 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.511862 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb2gj\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-kube-api-access-zb2gj\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.512270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.511941 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.512270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.511969 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.512270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.512038 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.512270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.512089 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.512270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.512198 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.512685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.512289 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.512685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.512364 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.512685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.512422 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.512863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.512833 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-certs\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.513344 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.513320 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-data\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.514250 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.514211 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-credential-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.514386 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.514348 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istiod-ca-cert\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.514615 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.514592 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-socket\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.524305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.524280 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-podinfo\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.532389 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.531994 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-envoy\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.569752 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.569717 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-token\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.570019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.569993 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb2gj\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-kube-api-access-zb2gj\") pod \"e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:21.656120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:21.656036 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:22.109880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:22.109855 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp"] Oct 31 15:10:22.110580 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:10:22.110560 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4efc5957_2d8d_4a14_b4fb_8d6404ef3f61.slice/crio-5e40a74f256bdb3bd1324a2315827c4d52d1528aafbace15516aa6e8ade9c83d WatchSource:0}: Error finding container 5e40a74f256bdb3bd1324a2315827c4d52d1528aafbace15516aa6e8ade9c83d: Status 404 returned error can't find the container with id 5e40a74f256bdb3bd1324a2315827c4d52d1528aafbace15516aa6e8ade9c83d Oct 31 15:10:22.113099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:22.113071 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:10:22.113178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:22.113131 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:10:22.113178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:22.113159 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:10:22.169991 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:22.169967 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" event={"ID":"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61","Type":"ContainerStarted","Data":"5e40a74f256bdb3bd1324a2315827c4d52d1528aafbace15516aa6e8ade9c83d"} Oct 31 15:10:23.178786 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:23.178744 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" event={"ID":"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61","Type":"ContainerStarted","Data":"5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af"} Oct 31 15:10:23.344355 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:23.344289 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" podStartSLOduration=2.344275077 podStartE2EDuration="2.344275077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:10:23.292490982 +0000 UTC m=+1466.503732400" watchObservedRunningTime="2026-10-31 15:10:23.344275077 +0000 UTC m=+1466.555516489" Oct 31 15:10:23.656562 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:23.656521 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:23.662208 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:23.662186 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:24.185194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:24.185171 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:24.186104 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:10:24.186082 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:10:26.208811 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:10:26.208777 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:10:56.218196 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:10:56.218149 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:11:04.338235 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:04.338202 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr"] Oct 31 15:11:09.339114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.339071 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" podUID="b15498a0-5e44-465f-af26-41b5130d5bfc" containerName="payload-pre-processing" containerID="cri-o://1709f464086e4102b65ce63cf195525c47407c6d9bc84fd85e914c647f508f79" gracePeriod=25 Oct 31 15:11:09.532028 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.531938 4399 generic.go:356] "Generic (PLEG): container finished" podID="b15498a0-5e44-465f-af26-41b5130d5bfc" containerID="1709f464086e4102b65ce63cf195525c47407c6d9bc84fd85e914c647f508f79" exitCode=0 Oct 31 15:11:09.532028 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.531977 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" event={"ID":"b15498a0-5e44-465f-af26-41b5130d5bfc","Type":"ContainerDied","Data":"1709f464086e4102b65ce63cf195525c47407c6d9bc84fd85e914c647f508f79"} Oct 31 15:11:09.621994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.621973 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:11:09.765646 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.765611 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b15498a0-5e44-465f-af26-41b5130d5bfc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15498a0-5e44-465f-af26-41b5130d5bfc-plugins-config-volume\") pod \"b15498a0-5e44-465f-af26-41b5130d5bfc\" (UID: \"b15498a0-5e44-465f-af26-41b5130d5bfc\") " Oct 31 15:11:09.765856 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.765735 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b15498a0-5e44-465f-af26-41b5130d5bfc-kube-api-access-ld74f\" (UniqueName: \"kubernetes.io/projected/b15498a0-5e44-465f-af26-41b5130d5bfc-kube-api-access-ld74f\") pod \"b15498a0-5e44-465f-af26-41b5130d5bfc\" (UID: \"b15498a0-5e44-465f-af26-41b5130d5bfc\") " Oct 31 15:11:09.766059 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.766035 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b15498a0-5e44-465f-af26-41b5130d5bfc-plugins-config-volume" pod "b15498a0-5e44-465f-af26-41b5130d5bfc" (UID: "b15498a0-5e44-465f-af26-41b5130d5bfc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:09.766354 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.766338 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b15498a0-5e44-465f-af26-41b5130d5bfc-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:09.769029 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.768995 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b15498a0-5e44-465f-af26-41b5130d5bfc-kube-api-access-ld74f" pod "b15498a0-5e44-465f-af26-41b5130d5bfc" (UID: "b15498a0-5e44-465f-af26-41b5130d5bfc"). InnerVolumeSpecName "kube-api-access-ld74f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:09.868291 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:09.868211 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld74f\" (UniqueName: \"kubernetes.io/projected/b15498a0-5e44-465f-af26-41b5130d5bfc-kube-api-access-ld74f\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:10.540040 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:10.539999 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" event={"ID":"b15498a0-5e44-465f-af26-41b5130d5bfc","Type":"ContainerDied","Data":"062a7c9454268e92cf1759e58cca7f60743822fdaa176e7a66aa117b320bb67f"} Oct 31 15:11:10.540040 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:10.540023 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr" Oct 31 15:11:10.540542 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:10.540059 4399 scope.go:122] "RemoveContainer" containerID="1709f464086e4102b65ce63cf195525c47407c6d9bc84fd85e914c647f508f79" Oct 31 15:11:10.704217 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:10.704185 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr"] Oct 31 15:11:10.752663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:10.752634 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf5d51-8569c487c5-vm6jr"] Oct 31 15:11:10.898890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:10.898818 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b15498a0-5e44-465f-af26-41b5130d5bfc" path="/var/lib/kubelet/pods/b15498a0-5e44-465f-af26-41b5130d5bfc/volumes" Oct 31 15:11:15.759312 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.759281 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw"] Oct 31 15:11:15.759982 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.759967 4399 cpu_manager.go:414] "containerMap: removing container" podUID="b15498a0-5e44-465f-af26-41b5130d5bfc" containerName="payload-pre-processing" Oct 31 15:11:15.760038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.759983 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b15498a0-5e44-465f-af26-41b5130d5bfc" containerName="payload-pre-processing" Oct 31 15:11:15.760113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.760103 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="b15498a0-5e44-465f-af26-41b5130d5bfc" containerName="payload-pre-processing" Oct 31 15:11:15.765008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.764993 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:15.820671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.820633 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw"] Oct 31 15:11:15.865663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.865625 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-644a8a-serving-cert\"" Oct 31 15:11:15.954936 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.954894 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cf25\" (UniqueName: \"kubernetes.io/projected/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-kube-api-access-9cf25\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:15.955138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.955042 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:15.955138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.955101 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:15.955278 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:15.955144 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-metrics-certs\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.057115 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.057018 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cf25\" (UniqueName: \"kubernetes.io/projected/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-kube-api-access-9cf25\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.057535 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.057463 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.057535 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.057493 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.057940 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.057559 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-metrics-certs\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.058556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.058534 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.060978 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.060953 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-metrics-certs\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.061489 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.061087 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.114046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.114022 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cf25\" (UniqueName: \"kubernetes.io/projected/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-kube-api-access-9cf25\") pod \"maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.378197 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.378097 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:16.758365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.758339 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw"] Oct 31 15:11:16.759576 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:11:16.759546 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7cc0df2_9462_47b4_96f0_6107c0cdc28e.slice/crio-502dee6db943056a61f558c050b400820da34fa981e79c74586b4e804362ed9c WatchSource:0}: Error finding container 502dee6db943056a61f558c050b400820da34fa981e79c74586b4e804362ed9c: Status 404 returned error can't find the container with id 502dee6db943056a61f558c050b400820da34fa981e79c74586b4e804362ed9c Oct 31 15:11:16.761250 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:16.761232 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 15:11:17.593996 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:17.593954 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" event={"ID":"b7cc0df2-9462-47b4-96f0-6107c0cdc28e","Type":"ContainerStarted","Data":"502dee6db943056a61f558c050b400820da34fa981e79c74586b4e804362ed9c"} Oct 31 15:11:18.601571 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:18.601533 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" event={"ID":"b7cc0df2-9462-47b4-96f0-6107c0cdc28e","Type":"ContainerStarted","Data":"2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872"} Oct 31 15:11:18.601993 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:18.601667 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:18.718153 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:18.718104 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" podStartSLOduration=2.897638268 podStartE2EDuration="3.718090435s" totalImagesPullingTime="820.452167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:18.714649404 +0000 UTC m=+1521.925890816" watchObservedRunningTime="2026-10-31 15:11:18.718090435 +0000 UTC m=+1521.929331847" Oct 31 15:11:24.613099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:24.613064 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:11:26.207397 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:11:26.207361 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:11:37.195386 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.195352 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm"] Oct 31 15:11:37.265813 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.265781 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm"] Oct 31 15:11:37.265978 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.265937 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.336585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.336560 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5f17a1-openshift-default-dockercfg-96fx9\"" Oct 31 15:11:37.362455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.362421 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.362619 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.362536 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.362619 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.362574 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.362722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.362657 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.362777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.362767 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.362838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.362826 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv87c\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-kube-api-access-rv87c\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.362885 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.362859 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954d2d01-a14b-40b4-8f26-469bf2909f91-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.362954 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.362941 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.362991 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.362966 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.464497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464409 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.464666 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464516 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.464666 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464543 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.464666 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464605 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.464896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464681 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.464896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464780 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv87c\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-kube-api-access-rv87c\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.464896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464825 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954d2d01-a14b-40b4-8f26-469bf2909f91-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.464896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464858 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.464896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464869 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.465152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464910 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.465152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.464920 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.465255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.465147 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.465255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.465169 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.465970 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.465946 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954d2d01-a14b-40b4-8f26-469bf2909f91-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.467254 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.467227 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.467374 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.467351 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.520571 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.520543 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.520863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.520844 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv87c\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-kube-api-access-rv87c\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:37.605872 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:37.605839 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:38.076002 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:11:38.075974 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954d2d01_a14b_40b4_8f26_469bf2909f91.slice/crio-6567a3a9b3b00ec3da0c11ab93daec4b62ac39f150847b5f241295743e67d1bb WatchSource:0}: Error finding container 6567a3a9b3b00ec3da0c11ab93daec4b62ac39f150847b5f241295743e67d1bb: Status 404 returned error can't find the container with id 6567a3a9b3b00ec3da0c11ab93daec4b62ac39f150847b5f241295743e67d1bb Oct 31 15:11:38.077852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:38.077732 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm"] Oct 31 15:11:38.078681 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:38.078648 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:11:38.078818 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:38.078731 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:11:38.078818 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:38.078761 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:11:38.874245 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:38.874201 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" event={"ID":"954d2d01-a14b-40b4-8f26-469bf2909f91","Type":"ContainerStarted","Data":"762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9"} Oct 31 15:11:38.874762 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:38.874278 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" event={"ID":"954d2d01-a14b-40b4-8f26-469bf2909f91","Type":"ContainerStarted","Data":"6567a3a9b3b00ec3da0c11ab93daec4b62ac39f150847b5f241295743e67d1bb"} Oct 31 15:11:39.091252 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.091193 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" podStartSLOduration=2.091179571 podStartE2EDuration="2.091179571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:11:39.040050243 +0000 UTC m=+1542.251291656" watchObservedRunningTime="2026-10-31 15:11:39.091179571 +0000 UTC m=+1542.302420982" Oct 31 15:11:39.606164 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.606135 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:39.609528 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.608213 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:11:39.609528 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.608255 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 15:11:39.715065 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.715038 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": read tcp 10.133.0.2:37702->10.133.0.86:15021: read: connection reset by peer" start-of-body= Oct 31 15:11:39.715224 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.715086 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": read tcp 10.133.0.2:37702->10.133.0.86:15021: read: connection reset by peer" Oct 31 15:11:39.882462 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.882340 4399 generic.go:356] "Generic (PLEG): container finished" podID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerID="5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af" exitCode=0 Oct 31 15:11:39.882462 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.882427 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" event={"ID":"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61","Type":"ContainerDied","Data":"5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af"} Oct 31 15:11:39.882955 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.882940 4399 scope.go:122] "RemoveContainer" containerID="5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af" Oct 31 15:11:39.884508 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.884484 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:11:39.884566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.884546 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:11:39.884605 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:39.884577 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:11:40.607244 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:40.607206 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:11:40.607431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:40.607257 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 15:11:40.901726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:40.901629 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" event={"ID":"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61","Type":"ContainerStarted","Data":"f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3"} Oct 31 15:11:41.606974 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:41.606939 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:11:41.607263 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:41.607234 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 15:11:41.666592 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:41.666564 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:11:41.666927 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:41.666809 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:11:41.667102 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:41.667080 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 15:11:42.634858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:42.634820 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:11:42.691533 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:42.638786 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 15:11:42.698725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:42.694391 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:11:42.698992 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:42.698961 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 15:11:43.608742 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:43.608240 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 31 15:11:43.608742 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:43.608304 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 31 15:11:43.681729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:43.676657 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:11:43.681729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:43.676735 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 15:11:44.611901 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:44.611864 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:44.612171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:44.612149 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:44.613075 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:44.613058 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:11:44.661638 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:44.661122 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:11:44.661638 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:44.661221 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 15:11:45.662142 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:45.662100 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Oct 31 15:11:45.662795 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:45.662760 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Oct 31 15:11:46.661257 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:46.661231 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:11:46.661664 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:46.661644 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:11:46.662511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:46.662489 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:11:56.213209 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:11:56.213166 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:11:57.712504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:57.712464 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk"] Oct 31 15:11:57.713058 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:57.712748 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" podUID="e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" containerName="maas-api" containerID="cri-o://bc2c6d46bfb1aa470313acdfc3c878586365ebbaad204d1def354bedc5d0206d" gracePeriod=30 Oct 31 15:11:58.120195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.120044 4399 generic.go:356] "Generic (PLEG): container finished" podID="e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" containerID="bc2c6d46bfb1aa470313acdfc3c878586365ebbaad204d1def354bedc5d0206d" exitCode=0 Oct 31 15:11:58.120195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.120109 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" event={"ID":"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0","Type":"ContainerDied","Data":"bc2c6d46bfb1aa470313acdfc3c878586365ebbaad204d1def354bedc5d0206d"} Oct 31 15:11:58.326137 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.326105 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:11:58.398029 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.397998 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls\") pod \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " Oct 31 15:11:58.398236 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.398139 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-openshift-service-ca\") pod \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " Oct 31 15:11:58.398312 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.398296 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-kube-api-access-9xz79\" (UniqueName: \"kubernetes.io/projected/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-kube-api-access-9xz79\") pod \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " Oct 31 15:11:58.398396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.398372 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-metrics-certs\") pod \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\" (UID: \"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0\") " Oct 31 15:11:58.398534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.398504 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-openshift-service-ca" pod "e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" (UID: "e1ea3563-2e5b-492b-bcd2-1b6caaf997f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:11:58.399144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.399081 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:58.400863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.400837 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-kube-api-access-9xz79" pod "e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" (UID: "e1ea3563-2e5b-492b-bcd2-1b6caaf997f0"). InnerVolumeSpecName "kube-api-access-9xz79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:11:58.401236 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.401217 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-metrics-certs" pod "e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" (UID: "e1ea3563-2e5b-492b-bcd2-1b6caaf997f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:11:58.402457 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.401691 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls" pod "e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" (UID: "e1ea3563-2e5b-492b-bcd2-1b6caaf997f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:11:58.501359 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.501313 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xz79\" (UniqueName: \"kubernetes.io/projected/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-kube-api-access-9xz79\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:58.501359 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.501358 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:58.501359 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:58.501369 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:11:59.129133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:59.129107 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" Oct 31 15:11:59.129536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:59.129137 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk" event={"ID":"e1ea3563-2e5b-492b-bcd2-1b6caaf997f0","Type":"ContainerDied","Data":"82ff7a03b607144bdd1d9e192648ce7a26b69e11bd3944a157f1abdc1c0a1968"} Oct 31 15:11:59.129536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:59.129208 4399 scope.go:122] "RemoveContainer" containerID="bc2c6d46bfb1aa470313acdfc3c878586365ebbaad204d1def354bedc5d0206d" Oct 31 15:11:59.292874 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:59.292844 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk"] Oct 31 15:11:59.296084 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:11:59.296065 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cf5d51-768874c8b5-22ttk"] Oct 31 15:12:00.898174 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:00.898140 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" path="/var/lib/kubelet/pods/e1ea3563-2e5b-492b-bcd2-1b6caaf997f0/volumes" Oct 31 15:12:03.449507 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:03.449477 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r"] Oct 31 15:12:03.449994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:03.449737 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" containerID="cri-o://d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6" gracePeriod=30 Oct 31 15:12:03.572103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:03.572068 4399 patch_prober.go:29] interesting pod/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:12:03.572265 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:03.572136 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:12:08.759557 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.759532 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:12:08.857603 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.857575 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-socket\") pod \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " Oct 31 15:12:08.857820 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.857647 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-kube-api-access-lcf99\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-kube-api-access-lcf99\") pod \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " Oct 31 15:12:08.857820 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.857811 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-credential-socket\") pod \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " Oct 31 15:12:08.857933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.857856 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-socket" pod "6494ac2c-f5da-4c61-8d14-b97acfe5aedf" (UID: "6494ac2c-f5da-4c61-8d14-b97acfe5aedf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:08.857933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.857867 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istiod-ca-cert\") pod \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " Oct 31 15:12:08.857933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.857907 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-token\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-token\") pod \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " Oct 31 15:12:08.858046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.857961 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-envoy\") pod \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " Oct 31 15:12:08.858046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.857997 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-data\") pod \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " Oct 31 15:12:08.858046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.858040 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-certs\") pod \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " Oct 31 15:12:08.858206 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.858057 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-credential-socket" pod "6494ac2c-f5da-4c61-8d14-b97acfe5aedf" (UID: "6494ac2c-f5da-4c61-8d14-b97acfe5aedf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:08.858206 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.858102 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-podinfo\") pod \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\" (UID: \"6494ac2c-f5da-4c61-8d14-b97acfe5aedf\") " Oct 31 15:12:08.858310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.858196 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istiod-ca-cert" pod "6494ac2c-f5da-4c61-8d14-b97acfe5aedf" (UID: "6494ac2c-f5da-4c61-8d14-b97acfe5aedf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:08.858364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.858325 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-data" pod "6494ac2c-f5da-4c61-8d14-b97acfe5aedf" (UID: "6494ac2c-f5da-4c61-8d14-b97acfe5aedf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:08.858417 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.858353 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-certs" pod "6494ac2c-f5da-4c61-8d14-b97acfe5aedf" (UID: "6494ac2c-f5da-4c61-8d14-b97acfe5aedf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:08.859046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.858977 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:08.859046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.859011 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:08.859046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.859028 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:08.859046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.859043 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:08.859046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.859057 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:08.860173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.860151 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-kube-api-access-lcf99" pod "6494ac2c-f5da-4c61-8d14-b97acfe5aedf" (UID: "6494ac2c-f5da-4c61-8d14-b97acfe5aedf"). InnerVolumeSpecName "kube-api-access-lcf99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:08.860282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.860181 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-token" pod "6494ac2c-f5da-4c61-8d14-b97acfe5aedf" (UID: "6494ac2c-f5da-4c61-8d14-b97acfe5aedf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:08.860282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.860208 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-envoy" pod "6494ac2c-f5da-4c61-8d14-b97acfe5aedf" (UID: "6494ac2c-f5da-4c61-8d14-b97acfe5aedf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:08.860560 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.860538 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-podinfo" pod "6494ac2c-f5da-4c61-8d14-b97acfe5aedf" (UID: "6494ac2c-f5da-4c61-8d14-b97acfe5aedf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:08.960346 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.960313 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:08.960346 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.960348 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:08.960525 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.960363 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:08.960525 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:08.960378 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcf99\" (UniqueName: \"kubernetes.io/projected/6494ac2c-f5da-4c61-8d14-b97acfe5aedf-kube-api-access-lcf99\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:09.257817 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.257785 4399 generic.go:356] "Generic (PLEG): container finished" podID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerID="d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6" exitCode=0 Oct 31 15:12:09.257988 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.257838 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" Oct 31 15:12:09.257988 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.257848 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" event={"ID":"6494ac2c-f5da-4c61-8d14-b97acfe5aedf","Type":"ContainerDied","Data":"d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6"} Oct 31 15:12:09.257988 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.257903 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r" event={"ID":"6494ac2c-f5da-4c61-8d14-b97acfe5aedf","Type":"ContainerDied","Data":"4d5705b76046f71180f84886e9760b1939fa71e8e9c4ce419d79e5c040766f10"} Oct 31 15:12:09.257988 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.257925 4399 scope.go:122] "RemoveContainer" containerID="d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6" Oct 31 15:12:09.268660 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.268644 4399 scope.go:122] "RemoveContainer" containerID="0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39" Oct 31 15:12:09.278159 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.278142 4399 scope.go:122] "RemoveContainer" containerID="d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6" Oct 31 15:12:09.278424 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:09.278403 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6\": container with ID starting with d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6 not found: ID does not exist" containerID="d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6" Oct 31 15:12:09.278477 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.278434 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6"} err="failed to get container status \"d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6\": rpc error: code = NotFound desc = could not find container \"d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6\": container with ID starting with d63be1339f40d1e064ecc68d4c5826563fd5e8cdbd63b27c019f83c65cb980d6 not found: ID does not exist" Oct 31 15:12:09.278477 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.278451 4399 scope.go:122] "RemoveContainer" containerID="0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39" Oct 31 15:12:09.278684 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:09.278667 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39\": container with ID starting with 0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39 not found: ID does not exist" containerID="0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39" Oct 31 15:12:09.278759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.278691 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39"} err="failed to get container status \"0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39\": rpc error: code = NotFound desc = could not find container \"0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39\": container with ID starting with 0bdfacac9469c3d884fea68faaee0b9bfe8e57e8a31afa58dd6978aedf7daf39 not found: ID does not exist" Oct 31 15:12:09.436956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.436919 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r"] Oct 31 15:12:09.486873 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:09.486844 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cf5d51-openshift-default-59456c99b7-sz47r"] Oct 31 15:12:10.898311 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:10.898273 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" path="/var/lib/kubelet/pods/6494ac2c-f5da-4c61-8d14-b97acfe5aedf/volumes" Oct 31 15:12:13.034607 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:13.034567 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd"] Oct 31 15:12:13.151322 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:13.151289 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8"] Oct 31 15:12:15.302881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.302850 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn"] Oct 31 15:12:15.303625 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.303610 4399 cpu_manager.go:414] "containerMap: removing container" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" Oct 31 15:12:15.303670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.303627 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" Oct 31 15:12:15.303670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.303659 4399 cpu_manager.go:414] "containerMap: removing container" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" Oct 31 15:12:15.303670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.303664 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" Oct 31 15:12:15.303837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.303676 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" containerName="maas-api" Oct 31 15:12:15.303837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.303682 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" containerName="maas-api" Oct 31 15:12:15.303837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.303798 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" Oct 31 15:12:15.303837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.303812 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ea3563-2e5b-492b-bcd2-1b6caaf997f0" containerName="maas-api" Oct 31 15:12:15.308834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.308815 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.364171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.364143 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn"] Oct 31 15:12:15.369449 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.369422 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7ftn\" (UniqueName: \"kubernetes.io/projected/98694ba7-d718-4754-aec2-be70a73ddfaf-kube-api-access-l7ftn\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.369678 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.369641 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.369754 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.369724 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-metrics-certs\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.369820 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.369800 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98694ba7-d718-4754-aec2-be70a73ddfaf-openshift-service-ca\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.408075 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.408029 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-5695787b-serving-cert\"" Oct 31 15:12:15.470891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.470856 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7ftn\" (UniqueName: \"kubernetes.io/projected/98694ba7-d718-4754-aec2-be70a73ddfaf-kube-api-access-l7ftn\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.471066 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.470961 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.471066 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.470985 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-metrics-certs\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.471066 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.471025 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98694ba7-d718-4754-aec2-be70a73ddfaf-openshift-service-ca\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.471066 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:15.471060 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-serving-cert: secret "maas-api-e2e-mt-5695787b-serving-cert" not found Oct 31 15:12:15.471314 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:15.471120 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls podName:98694ba7-d718-4754-aec2-be70a73ddfaf nodeName:}" failed. No retries permitted until 2026-10-31 15:12:15.971105459 +0000 UTC m=+1579.182346848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls") pod "maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" (UID: "98694ba7-d718-4754-aec2-be70a73ddfaf") : secret "maas-api-e2e-mt-5695787b-serving-cert" not found Oct 31 15:12:15.471811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.471792 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98694ba7-d718-4754-aec2-be70a73ddfaf-openshift-service-ca\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.473791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.473772 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-metrics-certs\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.532177 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.532142 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7ftn\" (UniqueName: \"kubernetes.io/projected/98694ba7-d718-4754-aec2-be70a73ddfaf-kube-api-access-l7ftn\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.981203 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.981167 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:15.983842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:15.983816 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls\") pod \"maas-api-e2e-mt-5695787b-77d88cdb7d-768nn\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:16.223781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:16.223752 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:16.602627 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:16.602588 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn"] Oct 31 15:12:16.603853 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:12:16.603821 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98694ba7_d718_4754_aec2_be70a73ddfaf.slice/crio-c1bd5e6435a677a5512ac988e44ebeb91f881e90d5adc179101da6ad0e83fa55 WatchSource:0}: Error finding container c1bd5e6435a677a5512ac988e44ebeb91f881e90d5adc179101da6ad0e83fa55: Status 404 returned error can't find the container with id c1bd5e6435a677a5512ac988e44ebeb91f881e90d5adc179101da6ad0e83fa55 Oct 31 15:12:17.333103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:17.333055 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" event={"ID":"98694ba7-d718-4754-aec2-be70a73ddfaf","Type":"ContainerStarted","Data":"c1bd5e6435a677a5512ac988e44ebeb91f881e90d5adc179101da6ad0e83fa55"} Oct 31 15:12:18.035126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.035061 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" podUID="a4c6f555-76aa-41ee-a5f1-1829292fe076" containerName="payload-processing" containerID="cri-o://e310ca93f9547fc49d5645c83680c9ac301aa3ec32204f109dbc0ef75ded8431" gracePeriod=25 Oct 31 15:12:18.152249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.152214 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" podUID="dc557f5b-3084-49e7-baf6-18fb698ca7cd" containerName="payload-pre-processing" containerID="cri-o://c47ee95299cbbbf318df4461c09f148d6571ae5abd56ca061af1dc386fa529a1" gracePeriod=25 Oct 31 15:12:18.344324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.344287 4399 generic.go:356] "Generic (PLEG): container finished" podID="a4c6f555-76aa-41ee-a5f1-1829292fe076" containerID="e310ca93f9547fc49d5645c83680c9ac301aa3ec32204f109dbc0ef75ded8431" exitCode=0 Oct 31 15:12:18.344784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.344465 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" event={"ID":"a4c6f555-76aa-41ee-a5f1-1829292fe076","Type":"ContainerDied","Data":"e310ca93f9547fc49d5645c83680c9ac301aa3ec32204f109dbc0ef75ded8431"} Oct 31 15:12:18.344784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.344511 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" event={"ID":"a4c6f555-76aa-41ee-a5f1-1829292fe076","Type":"ContainerDied","Data":"b97765453ca58f764e8fadb8809f5e7b84a50e9825ad5f328eedfb14f03c8345"} Oct 31 15:12:18.344784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.344528 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b97765453ca58f764e8fadb8809f5e7b84a50e9825ad5f328eedfb14f03c8345" Oct 31 15:12:18.346926 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.346799 4399 generic.go:356] "Generic (PLEG): container finished" podID="dc557f5b-3084-49e7-baf6-18fb698ca7cd" containerID="c47ee95299cbbbf318df4461c09f148d6571ae5abd56ca061af1dc386fa529a1" exitCode=0 Oct 31 15:12:18.346926 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.346909 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" event={"ID":"dc557f5b-3084-49e7-baf6-18fb698ca7cd","Type":"ContainerDied","Data":"c47ee95299cbbbf318df4461c09f148d6571ae5abd56ca061af1dc386fa529a1"} Oct 31 15:12:18.350021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.349931 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" event={"ID":"98694ba7-d718-4754-aec2-be70a73ddfaf","Type":"ContainerStarted","Data":"645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3"} Oct 31 15:12:18.350833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.350775 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:18.352216 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.352196 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:12:18.415103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.415076 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4c6f555-76aa-41ee-a5f1-1829292fe076-kube-api-access-nwsrq\" (UniqueName: \"kubernetes.io/projected/a4c6f555-76aa-41ee-a5f1-1829292fe076-kube-api-access-nwsrq\") pod \"a4c6f555-76aa-41ee-a5f1-1829292fe076\" (UID: \"a4c6f555-76aa-41ee-a5f1-1829292fe076\") " Oct 31 15:12:18.415246 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.415172 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4c6f555-76aa-41ee-a5f1-1829292fe076-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c6f555-76aa-41ee-a5f1-1829292fe076-plugins-config-volume\") pod \"a4c6f555-76aa-41ee-a5f1-1829292fe076\" (UID: \"a4c6f555-76aa-41ee-a5f1-1829292fe076\") " Oct 31 15:12:18.415583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.415556 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c6f555-76aa-41ee-a5f1-1829292fe076-plugins-config-volume" pod "a4c6f555-76aa-41ee-a5f1-1829292fe076" (UID: "a4c6f555-76aa-41ee-a5f1-1829292fe076"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:18.416291 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.416263 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c6f555-76aa-41ee-a5f1-1829292fe076-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:18.433862 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.433835 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c6f555-76aa-41ee-a5f1-1829292fe076-kube-api-access-nwsrq" pod "a4c6f555-76aa-41ee-a5f1-1829292fe076" (UID: "a4c6f555-76aa-41ee-a5f1-1829292fe076"). InnerVolumeSpecName "kube-api-access-nwsrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:18.474596 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.474532 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" podStartSLOduration=2.518292173 podStartE2EDuration="3.474513112s" totalImagesPullingTime="956.220939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:18.473361912 +0000 UTC m=+1581.684603323" watchObservedRunningTime="2026-10-31 15:12:18.474513112 +0000 UTC m=+1581.685754528" Oct 31 15:12:18.475275 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.475251 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:12:18.517022 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.516997 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc557f5b-3084-49e7-baf6-18fb698ca7cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc557f5b-3084-49e7-baf6-18fb698ca7cd-plugins-config-volume\") pod \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\" (UID: \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\") " Oct 31 15:12:18.517177 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.517060 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc557f5b-3084-49e7-baf6-18fb698ca7cd-kube-api-access-sxt8c\" (UniqueName: \"kubernetes.io/projected/dc557f5b-3084-49e7-baf6-18fb698ca7cd-kube-api-access-sxt8c\") pod \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\" (UID: \"dc557f5b-3084-49e7-baf6-18fb698ca7cd\") " Oct 31 15:12:18.517441 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.517416 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc557f5b-3084-49e7-baf6-18fb698ca7cd-plugins-config-volume" pod "dc557f5b-3084-49e7-baf6-18fb698ca7cd" (UID: "dc557f5b-3084-49e7-baf6-18fb698ca7cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:18.517809 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.517777 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc557f5b-3084-49e7-baf6-18fb698ca7cd-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:18.517809 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.517807 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwsrq\" (UniqueName: \"kubernetes.io/projected/a4c6f555-76aa-41ee-a5f1-1829292fe076-kube-api-access-nwsrq\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:18.519350 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.519326 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc557f5b-3084-49e7-baf6-18fb698ca7cd-kube-api-access-sxt8c" pod "dc557f5b-3084-49e7-baf6-18fb698ca7cd" (UID: "dc557f5b-3084-49e7-baf6-18fb698ca7cd"). InnerVolumeSpecName "kube-api-access-sxt8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:18.640109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:18.640026 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxt8c\" (UniqueName: \"kubernetes.io/projected/dc557f5b-3084-49e7-baf6-18fb698ca7cd-kube-api-access-sxt8c\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:19.357728 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:19.357672 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" Oct 31 15:12:19.357728 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:19.357713 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8" event={"ID":"dc557f5b-3084-49e7-baf6-18fb698ca7cd","Type":"ContainerDied","Data":"8c5b213d3164b36a1b142c14d18a6d0ba74feaa48547d11237102157195e51ec"} Oct 31 15:12:19.358270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:19.357780 4399 scope.go:122] "RemoveContainer" containerID="c47ee95299cbbbf318df4461c09f148d6571ae5abd56ca061af1dc386fa529a1" Oct 31 15:12:19.358270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:19.358126 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd" Oct 31 15:12:19.523005 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:19.522947 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd"] Oct 31 15:12:19.526933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:19.526905 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4d3ea8-66bbbc5dc7-2rwvd"] Oct 31 15:12:19.651286 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:19.649772 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8"] Oct 31 15:12:19.652717 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:19.651529 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4d3ea8-64d8f64dff-wkll8"] Oct 31 15:12:20.897392 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:20.897349 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4c6f555-76aa-41ee-a5f1-1829292fe076" path="/var/lib/kubelet/pods/a4c6f555-76aa-41ee-a5f1-1829292fe076/volumes" Oct 31 15:12:20.898003 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:20.897982 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc557f5b-3084-49e7-baf6-18fb698ca7cd" path="/var/lib/kubelet/pods/dc557f5b-3084-49e7-baf6-18fb698ca7cd/volumes" Oct 31 15:12:22.530961 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.530926 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb"] Oct 31 15:12:22.531579 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.531564 4399 cpu_manager.go:414] "containerMap: removing container" podUID="a4c6f555-76aa-41ee-a5f1-1829292fe076" containerName="payload-processing" Oct 31 15:12:22.531579 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.531578 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4c6f555-76aa-41ee-a5f1-1829292fe076" containerName="payload-processing" Oct 31 15:12:22.531686 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.531591 4399 cpu_manager.go:414] "containerMap: removing container" podUID="dc557f5b-3084-49e7-baf6-18fb698ca7cd" containerName="payload-pre-processing" Oct 31 15:12:22.531686 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.531596 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc557f5b-3084-49e7-baf6-18fb698ca7cd" containerName="payload-pre-processing" Oct 31 15:12:22.531792 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.531734 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4c6f555-76aa-41ee-a5f1-1829292fe076" containerName="payload-processing" Oct 31 15:12:22.531792 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.531742 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc557f5b-3084-49e7-baf6-18fb698ca7cd" containerName="payload-pre-processing" Oct 31 15:12:22.531792 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.531754 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6494ac2c-f5da-4c61-8d14-b97acfe5aedf" containerName="istio-proxy" Oct 31 15:12:22.536756 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.536735 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.547378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.547356 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.547454 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.547389 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.547499 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.547474 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.547544 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.547528 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.547593 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.547574 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.547641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.547631 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.547776 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.547762 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.547821 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.547786 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.547921 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.547903 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkftw\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-kube-api-access-wkftw\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.607258 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.607224 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb"] Oct 31 15:12:22.649559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.649516 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkftw\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-kube-api-access-wkftw\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.649715 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.649618 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.649715 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.649651 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.649891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.649726 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.649891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.649785 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.649891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.649810 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.649891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.649880 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.650087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.650031 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.650087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.650063 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.650354 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.650330 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-certs\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.650354 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.650345 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-data\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.650715 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.650676 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istiod-ca-cert\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.650855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.650792 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-credential-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.650855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.650811 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-socket\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.657787 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.653716 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-envoy\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.657787 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.654122 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-podinfo\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.706750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.706716 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-token\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.706919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.706896 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkftw\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-kube-api-access-wkftw\") pod \"e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:22.850371 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:22.850300 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:23.336636 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.336611 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb"] Oct 31 15:12:23.338038 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:12:23.338013 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9461a39_85f3_43c8_a510_d8d8a184e9fa.slice/crio-ecfbe7ec753bbe4f7c0b2e4018b9fa9e694237ac76b1f9fa7f89e44346d91123 WatchSource:0}: Error finding container ecfbe7ec753bbe4f7c0b2e4018b9fa9e694237ac76b1f9fa7f89e44346d91123: Status 404 returned error can't find the container with id ecfbe7ec753bbe4f7c0b2e4018b9fa9e694237ac76b1f9fa7f89e44346d91123 Oct 31 15:12:23.341374 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.341343 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:12:23.341442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.341408 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:12:23.341487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.341441 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:12:23.387514 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.387486 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" event={"ID":"b9461a39-85f3-43c8-a510-d8d8a184e9fa","Type":"ContainerStarted","Data":"ecfbe7ec753bbe4f7c0b2e4018b9fa9e694237ac76b1f9fa7f89e44346d91123"} Oct 31 15:12:23.725808 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.725772 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9"] Oct 31 15:12:23.726252 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.726011 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" containerID="cri-o://c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4" gracePeriod=30 Oct 31 15:12:23.729883 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.729842 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4"] Oct 31 15:12:23.730100 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.730056 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" containerID="cri-o://a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa" gracePeriod=30 Oct 31 15:12:23.780462 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.777665 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw"] Oct 31 15:12:23.780462 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.777931 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" containerID="cri-o://412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432" gracePeriod=30 Oct 31 15:12:23.782296 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.782163 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf"] Oct 31 15:12:23.782471 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:23.782347 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" containerID="cri-o://3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d" gracePeriod=30 Oct 31 15:12:24.389389 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:24.388277 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:12:24.398490 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:24.398450 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" event={"ID":"b9461a39-85f3-43c8-a510-d8d8a184e9fa","Type":"ContainerStarted","Data":"284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703"} Oct 31 15:12:24.656779 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:24.647466 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" podStartSLOduration=2.64744502 podStartE2EDuration="2.64744502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:24.643543855 +0000 UTC m=+1587.854785269" watchObservedRunningTime="2026-10-31 15:12:24.64744502 +0000 UTC m=+1587.858686433" Oct 31 15:12:24.864174 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:24.863062 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:24.866194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:24.865867 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:12:24.866194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:24.865924 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 15:12:25.807077 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:25.807040 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:12:25.807247 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:25.807093 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:12:25.865869 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:25.865834 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:12:25.866245 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:25.865886 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 15:12:26.208057 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:26.208022 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:12:26.768548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:26.768499 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:12:26.768801 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:26.768575 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:12:26.854143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:26.854094 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:12:26.854421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:26.854383 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 15:12:27.861723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:27.858085 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:12:27.861723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:27.858145 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 15:12:28.667188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:28.667158 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:12:28.667350 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:28.667214 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:12:28.866408 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:28.866284 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body= Oct 31 15:12:28.866408 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:28.866353 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" Oct 31 15:12:28.912030 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:28.911040 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Oct 31 15:12:28.912030 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:28.911096 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Oct 31 15:12:29.173749 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.172179 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 15:12:29.174634 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.174567 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:12:29.285545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285511 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-certs\") pod \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " Oct 31 15:12:29.285644 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285561 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-istio-token\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-istio-token\") pod \"44eff05d-6c38-4c35-a488-5343be8cef30\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " Oct 31 15:12:29.285644 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285628 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-data\") pod \"44eff05d-6c38-4c35-a488-5343be8cef30\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " Oct 31 15:12:29.285731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285663 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-credential-socket\") pod \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " Oct 31 15:12:29.285764 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285740 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44eff05d-6c38-4c35-a488-5343be8cef30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eff05d-6c38-4c35-a488-5343be8cef30-istio-podinfo\") pod \"44eff05d-6c38-4c35-a488-5343be8cef30\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " Oct 31 15:12:29.285799 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285771 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-envoy\") pod \"44eff05d-6c38-4c35-a488-5343be8cef30\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " Oct 31 15:12:29.285838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285801 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-kube-api-access-qrnmd\") pod \"44eff05d-6c38-4c35-a488-5343be8cef30\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " Oct 31 15:12:29.285838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285829 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istiod-ca-cert\") pod \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " Oct 31 15:12:29.285900 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285871 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-envoy\") pod \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " Oct 31 15:12:29.285948 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285930 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-data\") pod \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " Oct 31 15:12:29.285983 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285964 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-kube-api-access-946w6\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-kube-api-access-946w6\") pod \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " Oct 31 15:12:29.286019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.285999 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-certs\") pod \"44eff05d-6c38-4c35-a488-5343be8cef30\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " Oct 31 15:12:29.286053 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.286028 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-credential-socket\") pod \"44eff05d-6c38-4c35-a488-5343be8cef30\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " Oct 31 15:12:29.286101 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.286088 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-socket\") pod \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " Oct 31 15:12:29.286238 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.286224 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44eff05d-6c38-4c35-a488-5343be8cef30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eff05d-6c38-4c35-a488-5343be8cef30-istiod-ca-cert\") pod \"44eff05d-6c38-4c35-a488-5343be8cef30\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " Oct 31 15:12:29.286319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.286306 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-token\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-token\") pod \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " Oct 31 15:12:29.286379 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.286363 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-podinfo\") pod \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\" (UID: \"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e\") " Oct 31 15:12:29.286414 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.286395 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-socket\") pod \"44eff05d-6c38-4c35-a488-5343be8cef30\" (UID: \"44eff05d-6c38-4c35-a488-5343be8cef30\") " Oct 31 15:12:29.287760 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.287645 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-socket" pod "44eff05d-6c38-4c35-a488-5343be8cef30" (UID: "44eff05d-6c38-4c35-a488-5343be8cef30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.288167 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.287979 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-certs" pod "44eff05d-6c38-4c35-a488-5343be8cef30" (UID: "44eff05d-6c38-4c35-a488-5343be8cef30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.288672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.288648 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44eff05d-6c38-4c35-a488-5343be8cef30-istiod-ca-cert" pod "44eff05d-6c38-4c35-a488-5343be8cef30" (UID: "44eff05d-6c38-4c35-a488-5343be8cef30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:29.288918 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.288656 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44eff05d-6c38-4c35-a488-5343be8cef30-istio-podinfo" pod "44eff05d-6c38-4c35-a488-5343be8cef30" (UID: "44eff05d-6c38-4c35-a488-5343be8cef30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:29.289181 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.289151 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-data" pod "44eff05d-6c38-4c35-a488-5343be8cef30" (UID: "44eff05d-6c38-4c35-a488-5343be8cef30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.289381 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.289362 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-credential-socket" pod "44eff05d-6c38-4c35-a488-5343be8cef30" (UID: "44eff05d-6c38-4c35-a488-5343be8cef30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.289483 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.289376 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-credential-socket" pod "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" (UID: "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.289629 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.289588 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-data" pod "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" (UID: "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.289722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.289385 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-kube-api-access-946w6" pod "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" (UID: "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e"). InnerVolumeSpecName "kube-api-access-946w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:29.289898 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.289788 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istiod-ca-cert" pod "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" (UID: "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:29.290025 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.289985 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-socket" pod "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" (UID: "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.290260 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.290242 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-token" pod "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" (UID: "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:29.290380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.290363 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-certs" pod "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" (UID: "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.290858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.290521 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-istio-token" pod "44eff05d-6c38-4c35-a488-5343be8cef30" (UID: "44eff05d-6c38-4c35-a488-5343be8cef30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:29.290858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.290548 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-podinfo" pod "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" (UID: "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:29.291879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.291856 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-envoy" pod "44eff05d-6c38-4c35-a488-5343be8cef30" (UID: "44eff05d-6c38-4c35-a488-5343be8cef30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.292094 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.292077 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-envoy" pod "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" (UID: "17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.292352 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.292331 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-kube-api-access-qrnmd" pod "44eff05d-6c38-4c35-a488-5343be8cef30" (UID: "44eff05d-6c38-4c35-a488-5343be8cef30"). InnerVolumeSpecName "kube-api-access-qrnmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:29.296210 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.296192 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:12:29.299182 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.299166 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:12:29.387633 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.387603 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-podinfo\") pod \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " Oct 31 15:12:29.387835 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.387645 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-envoy\") pod \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " Oct 31 15:12:29.387835 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.387674 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-certs\") pod \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " Oct 31 15:12:29.387835 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.387772 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-credential-socket\") pod \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " Oct 31 15:12:29.388005 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.387865 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-certs\") pod \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " Oct 31 15:12:29.388005 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.387911 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-envoy\") pod \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " Oct 31 15:12:29.388005 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.387940 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istiod-ca-cert\") pod \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " Oct 31 15:12:29.388005 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.387969 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-kube-api-access-97hgt\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-kube-api-access-97hgt\") pod \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " Oct 31 15:12:29.388212 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388086 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-token\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-token\") pod \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " Oct 31 15:12:29.388212 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388127 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istiod-ca-cert\") pod \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " Oct 31 15:12:29.388212 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388163 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-token\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-token\") pod \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " Oct 31 15:12:29.388481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388238 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-podinfo\") pod \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " Oct 31 15:12:29.388481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388299 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-kube-api-access-znww9\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-kube-api-access-znww9\") pod \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " Oct 31 15:12:29.388481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388349 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-socket\") pod \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " Oct 31 15:12:29.388481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388380 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-data\") pod \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " Oct 31 15:12:29.388481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388415 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-data\") pod \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " Oct 31 15:12:29.388481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388466 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-credential-socket\") pod \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\" (UID: \"270f3cc9-fae1-40bd-96aa-0451fd70d10e\") " Oct 31 15:12:29.388815 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388587 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-socket\") pod \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\" (UID: \"26232fdc-cf8d-4936-af6d-a8bd6a24e970\") " Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388158 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-certs" pod "26232fdc-cf8d-4936-af6d-a8bd6a24e970" (UID: "26232fdc-cf8d-4936-af6d-a8bd6a24e970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388243 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-credential-socket" pod "26232fdc-cf8d-4936-af6d-a8bd6a24e970" (UID: "26232fdc-cf8d-4936-af6d-a8bd6a24e970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388551 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-certs" pod "270f3cc9-fae1-40bd-96aa-0451fd70d10e" (UID: "270f3cc9-fae1-40bd-96aa-0451fd70d10e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.388575 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istiod-ca-cert" pod "26232fdc-cf8d-4936-af6d-a8bd6a24e970" (UID: "26232fdc-cf8d-4936-af6d-a8bd6a24e970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389084 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-data" pod "26232fdc-cf8d-4936-af6d-a8bd6a24e970" (UID: "26232fdc-cf8d-4936-af6d-a8bd6a24e970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389243 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-socket" pod "270f3cc9-fae1-40bd-96aa-0451fd70d10e" (UID: "270f3cc9-fae1-40bd-96aa-0451fd70d10e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389549 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-socket" pod "26232fdc-cf8d-4936-af6d-a8bd6a24e970" (UID: "26232fdc-cf8d-4936-af6d-a8bd6a24e970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389562 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-credential-socket" pod "270f3cc9-fae1-40bd-96aa-0451fd70d10e" (UID: "270f3cc9-fae1-40bd-96aa-0451fd70d10e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389577 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istiod-ca-cert" pod "270f3cc9-fae1-40bd-96aa-0451fd70d10e" (UID: "270f3cc9-fae1-40bd-96aa-0451fd70d10e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:29.389631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389635 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389658 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389673 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389689 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946w6\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-kube-api-access-946w6\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389726 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389742 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389756 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389770 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44eff05d-6c38-4c35-a488-5343be8cef30-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389784 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389797 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389809 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389822 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389835 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389848 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389859 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389873 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44eff05d-6c38-4c35-a488-5343be8cef30-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389886 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44eff05d-6c38-4c35-a488-5343be8cef30-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.390421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.389902 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrnmd\" (UniqueName: \"kubernetes.io/projected/44eff05d-6c38-4c35-a488-5343be8cef30-kube-api-access-qrnmd\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.391540 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.391363 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-podinfo" pod "26232fdc-cf8d-4936-af6d-a8bd6a24e970" (UID: "26232fdc-cf8d-4936-af6d-a8bd6a24e970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:29.391931 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.391714 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-data" pod "270f3cc9-fae1-40bd-96aa-0451fd70d10e" (UID: "270f3cc9-fae1-40bd-96aa-0451fd70d10e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.392407 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.392380 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-token" pod "26232fdc-cf8d-4936-af6d-a8bd6a24e970" (UID: "26232fdc-cf8d-4936-af6d-a8bd6a24e970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:29.392518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.392414 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-envoy" pod "26232fdc-cf8d-4936-af6d-a8bd6a24e970" (UID: "26232fdc-cf8d-4936-af6d-a8bd6a24e970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.392518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.392472 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-kube-api-access-97hgt" pod "270f3cc9-fae1-40bd-96aa-0451fd70d10e" (UID: "270f3cc9-fae1-40bd-96aa-0451fd70d10e"). InnerVolumeSpecName "kube-api-access-97hgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:29.393742 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.393619 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-kube-api-access-znww9" pod "26232fdc-cf8d-4936-af6d-a8bd6a24e970" (UID: "26232fdc-cf8d-4936-af6d-a8bd6a24e970"). InnerVolumeSpecName "kube-api-access-znww9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:29.393843 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.393766 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-podinfo" pod "270f3cc9-fae1-40bd-96aa-0451fd70d10e" (UID: "270f3cc9-fae1-40bd-96aa-0451fd70d10e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:29.393938 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.393917 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-token" pod "270f3cc9-fae1-40bd-96aa-0451fd70d10e" (UID: "270f3cc9-fae1-40bd-96aa-0451fd70d10e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:29.394351 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.394330 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-envoy" pod "270f3cc9-fae1-40bd-96aa-0451fd70d10e" (UID: "270f3cc9-fae1-40bd-96aa-0451fd70d10e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:29.432959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.432929 4399 generic.go:356] "Generic (PLEG): container finished" podID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerID="412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432" exitCode=0 Oct 31 15:12:29.433189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.432975 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" Oct 31 15:12:29.433189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.433026 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" event={"ID":"270f3cc9-fae1-40bd-96aa-0451fd70d10e","Type":"ContainerDied","Data":"412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432"} Oct 31 15:12:29.433189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.433133 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw" event={"ID":"270f3cc9-fae1-40bd-96aa-0451fd70d10e","Type":"ContainerDied","Data":"400e591f8db6c26d207390bcad7caa14d9157195447aa108158a0beaf2cf6f6d"} Oct 31 15:12:29.433189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.433171 4399 scope.go:122] "RemoveContainer" containerID="412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432" Oct 31 15:12:29.434809 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.434789 4399 generic.go:356] "Generic (PLEG): container finished" podID="44eff05d-6c38-4c35-a488-5343be8cef30" containerID="a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa" exitCode=0 Oct 31 15:12:29.435014 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.434994 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" Oct 31 15:12:29.435186 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.435152 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" event={"ID":"44eff05d-6c38-4c35-a488-5343be8cef30","Type":"ContainerDied","Data":"a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa"} Oct 31 15:12:29.435338 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.435219 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4" event={"ID":"44eff05d-6c38-4c35-a488-5343be8cef30","Type":"ContainerDied","Data":"d2dc06baa1976aee6e3db2671d2ed32f5d665fa38e427eb6abe0b221e0c11cf1"} Oct 31 15:12:29.436401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.436356 4399 generic.go:356] "Generic (PLEG): container finished" podID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerID="3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d" exitCode=0 Oct 31 15:12:29.436401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.436379 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" event={"ID":"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e","Type":"ContainerDied","Data":"3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d"} Oct 31 15:12:29.436585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.436423 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" event={"ID":"17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e","Type":"ContainerDied","Data":"f56c467e3755343162a842961f9e71eed77a3b6c7bfa760e0470e2e3a77e8aef"} Oct 31 15:12:29.436585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.436440 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf" Oct 31 15:12:29.439342 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.439318 4399 generic.go:356] "Generic (PLEG): container finished" podID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerID="c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4" exitCode=0 Oct 31 15:12:29.439423 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.439388 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" event={"ID":"26232fdc-cf8d-4936-af6d-a8bd6a24e970","Type":"ContainerDied","Data":"c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4"} Oct 31 15:12:29.439478 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.439422 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" Oct 31 15:12:29.439478 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.439428 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9" event={"ID":"26232fdc-cf8d-4936-af6d-a8bd6a24e970","Type":"ContainerDied","Data":"4274037ed29d955683fe025f59c152909b8fa19a214f728f8b0247881dcccd35"} Oct 31 15:12:29.445913 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.445895 4399 scope.go:122] "RemoveContainer" containerID="412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432" Oct 31 15:12:29.446191 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:29.446164 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432\": container with ID starting with 412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432 not found: ID does not exist" containerID="412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432" Oct 31 15:12:29.446294 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.446206 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432"} err="failed to get container status \"412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432\": rpc error: code = NotFound desc = could not find container \"412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432\": container with ID starting with 412a0169d7f33f7a45f2e14f8ab3358e8e6f186233410fbb704cf99d4776b432 not found: ID does not exist" Oct 31 15:12:29.446294 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.446227 4399 scope.go:122] "RemoveContainer" containerID="a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa" Oct 31 15:12:29.457930 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.457914 4399 scope.go:122] "RemoveContainer" containerID="a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa" Oct 31 15:12:29.458337 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:29.458229 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa\": container with ID starting with a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa not found: ID does not exist" containerID="a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa" Oct 31 15:12:29.458337 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.458255 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa"} err="failed to get container status \"a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa\": rpc error: code = NotFound desc = could not find container \"a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa\": container with ID starting with a1c71ae3276b581e9dd64b8cac62818de06b716eca136bdbc023c39c0e7f08fa not found: ID does not exist" Oct 31 15:12:29.458337 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.458271 4399 scope.go:122] "RemoveContainer" containerID="3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d" Oct 31 15:12:29.486602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.486570 4399 scope.go:122] "RemoveContainer" containerID="3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d" Oct 31 15:12:29.487022 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:29.487001 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d\": container with ID starting with 3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d not found: ID does not exist" containerID="3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d" Oct 31 15:12:29.487121 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.487033 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d"} err="failed to get container status \"3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d\": rpc error: code = NotFound desc = could not find container \"3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d\": container with ID starting with 3009a75eb3b84ed4b64bf1dc60c810d5ad3877b4de3e4d43497ffa3fcc62237d not found: ID does not exist" Oct 31 15:12:29.487121 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.487052 4399 scope.go:122] "RemoveContainer" containerID="c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4" Oct 31 15:12:29.491570 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491539 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.491677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491576 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.491677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491593 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.491677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491609 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.491677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491625 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97hgt\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-kube-api-access-97hgt\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.491677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491640 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.491677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491653 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.491677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491666 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.491677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491679 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.492099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491735 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znww9\" (UniqueName: \"kubernetes.io/projected/26232fdc-cf8d-4936-af6d-a8bd6a24e970-kube-api-access-znww9\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.492099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491751 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.492099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491765 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.492099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491778 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.492099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491790 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.492099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491804 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.492099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491815 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.492099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491829 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26232fdc-cf8d-4936-af6d-a8bd6a24e970-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.492099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.491842 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/270f3cc9-fae1-40bd-96aa-0451fd70d10e-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:29.505528 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.502058 4399 scope.go:122] "RemoveContainer" containerID="c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4" Oct 31 15:12:29.505528 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:29.503512 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4\": container with ID starting with c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4 not found: ID does not exist" containerID="c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4" Oct 31 15:12:29.505528 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.503545 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4"} err="failed to get container status \"c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4\": rpc error: code = NotFound desc = could not find container \"c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4\": container with ID starting with c3579d6c281087216f05830f3eaf9e0a70766c55154034e5247d5283b4df54b4 not found: ID does not exist" Oct 31 15:12:29.821815 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.821780 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw"] Oct 31 15:12:29.825416 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.825381 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-rssvw"] Oct 31 15:12:29.855454 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.855431 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:29.855838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.855819 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:29.856651 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.856634 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:12:29.934375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.934341 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf"] Oct 31 15:12:29.938644 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:29.938615 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-zlmzf"] Oct 31 15:12:30.044623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:30.044541 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9"] Oct 31 15:12:30.051072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:30.051046 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-xbjk9"] Oct 31 15:12:30.160901 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:30.160866 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4"] Oct 31 15:12:30.166955 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:30.166928 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-mmcf4"] Oct 31 15:12:30.897818 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:30.897788 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" path="/var/lib/kubelet/pods/17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e/volumes" Oct 31 15:12:30.898279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:30.898264 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" path="/var/lib/kubelet/pods/26232fdc-cf8d-4936-af6d-a8bd6a24e970/volumes" Oct 31 15:12:30.898690 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:30.898676 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" path="/var/lib/kubelet/pods/270f3cc9-fae1-40bd-96aa-0451fd70d10e/volumes" Oct 31 15:12:30.899469 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:30.899452 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" path="/var/lib/kubelet/pods/44eff05d-6c38-4c35-a488-5343be8cef30/volumes" Oct 31 15:12:31.454042 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454014 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z"] Oct 31 15:12:31.454675 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454661 4399 cpu_manager.go:414] "containerMap: removing container" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" Oct 31 15:12:31.454675 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454675 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" Oct 31 15:12:31.454833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454744 4399 cpu_manager.go:414] "containerMap: removing container" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" Oct 31 15:12:31.454833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454751 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" Oct 31 15:12:31.454833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454769 4399 cpu_manager.go:414] "containerMap: removing container" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" Oct 31 15:12:31.454833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454774 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" Oct 31 15:12:31.454833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454792 4399 cpu_manager.go:414] "containerMap: removing container" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" Oct 31 15:12:31.454833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454796 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" Oct 31 15:12:31.455062 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454899 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e6b5a0-a02a-4d6a-b190-7e1f7176fb3e" containerName="istio-proxy" Oct 31 15:12:31.455062 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454908 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="26232fdc-cf8d-4936-af6d-a8bd6a24e970" containerName="istio-proxy" Oct 31 15:12:31.455062 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454920 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="270f3cc9-fae1-40bd-96aa-0451fd70d10e" containerName="istio-proxy" Oct 31 15:12:31.455062 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.454927 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="44eff05d-6c38-4c35-a488-5343be8cef30" containerName="istio-proxy" Oct 31 15:12:31.460289 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.460269 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:31.526752 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.523121 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-94a889\"" Oct 31 15:12:31.529587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.529555 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z"] Oct 31 15:12:31.540885 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.539791 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0860d0-4608-4a18-a0a1-fed76403af03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z\" (UID: \"2d0860d0-4608-4a18-a0a1-fed76403af03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:31.540885 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.539864 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62c2\" (UniqueName: \"kubernetes.io/projected/2d0860d0-4608-4a18-a0a1-fed76403af03-kube-api-access-q62c2\") pod \"payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z\" (UID: \"2d0860d0-4608-4a18-a0a1-fed76403af03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:31.579182 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.579146 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-94a889-dockercfg-t4dzc\"" Oct 31 15:12:31.642099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.642060 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0860d0-4608-4a18-a0a1-fed76403af03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z\" (UID: \"2d0860d0-4608-4a18-a0a1-fed76403af03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:31.642318 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.642125 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62c2\" (UniqueName: \"kubernetes.io/projected/2d0860d0-4608-4a18-a0a1-fed76403af03-kube-api-access-q62c2\") pod \"payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z\" (UID: \"2d0860d0-4608-4a18-a0a1-fed76403af03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:31.643880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.643840 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0860d0-4608-4a18-a0a1-fed76403af03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z\" (UID: \"2d0860d0-4608-4a18-a0a1-fed76403af03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:31.655738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.655688 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs"] Oct 31 15:12:31.660730 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.660660 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:31.673569 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.673536 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs"] Oct 31 15:12:31.716126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.716054 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62c2\" (UniqueName: \"kubernetes.io/projected/2d0860d0-4608-4a18-a0a1-fed76403af03-kube-api-access-q62c2\") pod \"payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z\" (UID: \"2d0860d0-4608-4a18-a0a1-fed76403af03\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:31.755981 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.754900 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8czs\" (UniqueName: \"kubernetes.io/projected/78e95b96-7906-4dbe-8bd2-72d115517ae8-kube-api-access-j8czs\") pod \"payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs\" (UID: \"78e95b96-7906-4dbe-8bd2-72d115517ae8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:31.755981 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.754980 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e95b96-7906-4dbe-8bd2-72d115517ae8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs\" (UID: \"78e95b96-7906-4dbe-8bd2-72d115517ae8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:31.774302 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.774247 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:31.864978 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.856272 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e95b96-7906-4dbe-8bd2-72d115517ae8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs\" (UID: \"78e95b96-7906-4dbe-8bd2-72d115517ae8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:31.864978 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.856632 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8czs\" (UniqueName: \"kubernetes.io/projected/78e95b96-7906-4dbe-8bd2-72d115517ae8-kube-api-access-j8czs\") pod \"payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs\" (UID: \"78e95b96-7906-4dbe-8bd2-72d115517ae8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:31.864978 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.857803 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e95b96-7906-4dbe-8bd2-72d115517ae8-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs\" (UID: \"78e95b96-7906-4dbe-8bd2-72d115517ae8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:31.914036 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:31.914003 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8czs\" (UniqueName: \"kubernetes.io/projected/78e95b96-7906-4dbe-8bd2-72d115517ae8-kube-api-access-j8czs\") pod \"payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs\" (UID: \"78e95b96-7906-4dbe-8bd2-72d115517ae8\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:32.005729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:32.005624 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:32.157787 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:32.157710 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z"] Oct 31 15:12:32.391947 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:32.391918 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs"] Oct 31 15:12:32.393637 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:12:32.393607 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e95b96_7906_4dbe_8bd2_72d115517ae8.slice/crio-a00280f1ab1d28b5e6d8a2759931d6a4baa090a803b8afc47d949ce9f6d0c7fb WatchSource:0}: Error finding container a00280f1ab1d28b5e6d8a2759931d6a4baa090a803b8afc47d949ce9f6d0c7fb: Status 404 returned error can't find the container with id a00280f1ab1d28b5e6d8a2759931d6a4baa090a803b8afc47d949ce9f6d0c7fb Oct 31 15:12:32.468446 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:32.468411 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" event={"ID":"78e95b96-7906-4dbe-8bd2-72d115517ae8","Type":"ContainerStarted","Data":"a00280f1ab1d28b5e6d8a2759931d6a4baa090a803b8afc47d949ce9f6d0c7fb"} Oct 31 15:12:32.470286 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:32.470259 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" event={"ID":"2d0860d0-4608-4a18-a0a1-fed76403af03","Type":"ContainerStarted","Data":"b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3"} Oct 31 15:12:32.470401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:32.470321 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:32.470401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:32.470350 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" event={"ID":"2d0860d0-4608-4a18-a0a1-fed76403af03","Type":"ContainerStarted","Data":"15fc6577534ac4160735b6b7e0da7ffcccaa600114cf4d7f43a9adb4c44e96ab"} Oct 31 15:12:32.606791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:32.606677 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" podStartSLOduration=1.606664652 podStartE2EDuration="1.606664652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:32.604469921 +0000 UTC m=+1595.815711344" watchObservedRunningTime="2026-10-31 15:12:32.606664652 +0000 UTC m=+1595.817906065" Oct 31 15:12:33.478047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.478010 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" event={"ID":"78e95b96-7906-4dbe-8bd2-72d115517ae8","Type":"ContainerStarted","Data":"940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111"} Oct 31 15:12:33.478512 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.478315 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:33.594142 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.594094 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" podStartSLOduration=2.594081663 podStartE2EDuration="2.594081663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:33.590234896 +0000 UTC m=+1596.801476308" watchObservedRunningTime="2026-10-31 15:12:33.594081663 +0000 UTC m=+1596.805323076" Oct 31 15:12:33.649976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.649944 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb"] Oct 31 15:12:33.650162 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.650143 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" containerID="cri-o://ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc" gracePeriod=30 Oct 31 15:12:33.653230 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.653203 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d"] Oct 31 15:12:33.653421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.653396 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" containerID="cri-o://973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9" gracePeriod=30 Oct 31 15:12:33.702557 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.702519 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9"] Oct 31 15:12:33.703088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.703012 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" containerID="cri-o://eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c" gracePeriod=30 Oct 31 15:12:33.717245 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.717208 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp"] Oct 31 15:12:33.717599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:33.717561 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" containerID="cri-o://8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170" gracePeriod=30 Oct 31 15:12:36.636274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:36.636238 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:12:36.636787 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:36.636293 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:12:37.668510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:37.668469 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:12:37.669031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:37.668553 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:12:37.720545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:37.720510 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:12:37.720670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:37.720571 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:12:39.034556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.034531 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:12:39.056599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.056435 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:12:39.113341 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.113320 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:12:39.118547 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.118525 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:12:39.164710 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.164668 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-podinfo\") pod \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " Oct 31 15:12:39.164845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.164744 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-certs\") pod \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " Oct 31 15:12:39.164845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.164764 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-envoy\") pod \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " Oct 31 15:12:39.164845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.164827 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-envoy\") pod \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " Oct 31 15:12:39.165010 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.164858 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-certs\") pod \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " Oct 31 15:12:39.165010 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.164915 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-credential-socket\") pod \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " Oct 31 15:12:39.165010 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.164954 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-socket\") pod \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " Oct 31 15:12:39.165010 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.164998 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-credential-socket\") pod \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " Oct 31 15:12:39.165390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165042 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-certs" pod "2d977f9c-25e6-451c-ae90-a0bbc31b2efd" (UID: "2d977f9c-25e6-451c-ae90-a0bbc31b2efd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.165390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165081 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-podinfo\") pod \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " Oct 31 15:12:39.165390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165148 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-data\") pod \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " Oct 31 15:12:39.165390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165191 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istiod-ca-cert\") pod \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " Oct 31 15:12:39.165390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165263 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-token\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-token\") pod \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " Oct 31 15:12:39.165390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165298 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-data\") pod \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " Oct 31 15:12:39.165390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165357 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-socket\") pod \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " Oct 31 15:12:39.165390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165188 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-certs" pod "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" (UID: "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.165832 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165595 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-data" pod "2d977f9c-25e6-451c-ae90-a0bbc31b2efd" (UID: "2d977f9c-25e6-451c-ae90-a0bbc31b2efd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.165958 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.165898 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istiod-ca-cert" pod "2d977f9c-25e6-451c-ae90-a0bbc31b2efd" (UID: "2d977f9c-25e6-451c-ae90-a0bbc31b2efd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:39.166133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.166109 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-kube-api-access-krggw\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-kube-api-access-krggw\") pod \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " Oct 31 15:12:39.166263 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.166223 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-token\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-token\") pod \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " Oct 31 15:12:39.166324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.166269 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istiod-ca-cert\") pod \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\" (UID: \"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1\") " Oct 31 15:12:39.166324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.166294 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-socket" pod "2d977f9c-25e6-451c-ae90-a0bbc31b2efd" (UID: "2d977f9c-25e6-451c-ae90-a0bbc31b2efd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.166324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.166315 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-kube-api-access-hwmn8\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-kube-api-access-hwmn8\") pod \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\" (UID: \"2d977f9c-25e6-451c-ae90-a0bbc31b2efd\") " Oct 31 15:12:39.166598 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.166549 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-credential-socket" pod "2d977f9c-25e6-451c-ae90-a0bbc31b2efd" (UID: "2d977f9c-25e6-451c-ae90-a0bbc31b2efd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.167133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.167054 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.167133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.167076 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.167133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.167086 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.167133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.167095 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.167133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.167107 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.167133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.167118 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.167549 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.167275 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-credential-socket" pod "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" (UID: "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.167549 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.167367 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-socket" pod "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" (UID: "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.168550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.168148 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-data" pod "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" (UID: "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.168550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.168278 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-envoy" pod "2d977f9c-25e6-451c-ae90-a0bbc31b2efd" (UID: "2d977f9c-25e6-451c-ae90-a0bbc31b2efd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.168550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.168486 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istiod-ca-cert" pod "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" (UID: "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:39.169303 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.169271 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-kube-api-access-hwmn8" pod "2d977f9c-25e6-451c-ae90-a0bbc31b2efd" (UID: "2d977f9c-25e6-451c-ae90-a0bbc31b2efd"). InnerVolumeSpecName "kube-api-access-hwmn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.169641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.169604 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-token" pod "2d977f9c-25e6-451c-ae90-a0bbc31b2efd" (UID: "2d977f9c-25e6-451c-ae90-a0bbc31b2efd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.169833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.169807 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-podinfo" pod "2d977f9c-25e6-451c-ae90-a0bbc31b2efd" (UID: "2d977f9c-25e6-451c-ae90-a0bbc31b2efd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:39.170364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.170338 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-envoy" pod "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" (UID: "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.170554 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.170530 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-podinfo" pod "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" (UID: "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:39.171032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.171014 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-kube-api-access-krggw" pod "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" (UID: "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1"). InnerVolumeSpecName "kube-api-access-krggw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.171632 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.171615 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-token" pod "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" (UID: "5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.267933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.267900 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-token\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-token\") pod \"be8f2787-9798-4c21-ae4e-7b1136b8b592\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " Oct 31 15:12:39.268105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.267966 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-kube-api-access-24x2d\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-kube-api-access-24x2d\") pod \"be8f2787-9798-4c21-ae4e-7b1136b8b592\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " Oct 31 15:12:39.268105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.267998 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-podinfo\") pod \"a8392178-c8ba-4b53-9cec-fe944f3eac47\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " Oct 31 15:12:39.268105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268030 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-envoy\") pod \"be8f2787-9798-4c21-ae4e-7b1136b8b592\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " Oct 31 15:12:39.268264 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268115 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-socket\") pod \"a8392178-c8ba-4b53-9cec-fe944f3eac47\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " Oct 31 15:12:39.268264 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268164 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-envoy\") pod \"a8392178-c8ba-4b53-9cec-fe944f3eac47\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " Oct 31 15:12:39.268264 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268224 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-data\") pod \"a8392178-c8ba-4b53-9cec-fe944f3eac47\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " Oct 31 15:12:39.268264 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268253 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-podinfo\") pod \"be8f2787-9798-4c21-ae4e-7b1136b8b592\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " Oct 31 15:12:39.268464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268312 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8392178-c8ba-4b53-9cec-fe944f3eac47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8392178-c8ba-4b53-9cec-fe944f3eac47-istiod-ca-cert\") pod \"a8392178-c8ba-4b53-9cec-fe944f3eac47\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " Oct 31 15:12:39.268464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268355 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-data\") pod \"be8f2787-9798-4c21-ae4e-7b1136b8b592\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " Oct 31 15:12:39.268464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268424 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8f2787-9798-4c21-ae4e-7b1136b8b592-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f2787-9798-4c21-ae4e-7b1136b8b592-istiod-ca-cert\") pod \"be8f2787-9798-4c21-ae4e-7b1136b8b592\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " Oct 31 15:12:39.268612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268460 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-socket" pod "a8392178-c8ba-4b53-9cec-fe944f3eac47" (UID: "a8392178-c8ba-4b53-9cec-fe944f3eac47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.268612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268493 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-certs\") pod \"a8392178-c8ba-4b53-9cec-fe944f3eac47\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " Oct 31 15:12:39.268612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268543 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-certs\") pod \"be8f2787-9798-4c21-ae4e-7b1136b8b592\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " Oct 31 15:12:39.268612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268573 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-credential-socket\") pod \"be8f2787-9798-4c21-ae4e-7b1136b8b592\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " Oct 31 15:12:39.268833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268628 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-token\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-token\") pod \"a8392178-c8ba-4b53-9cec-fe944f3eac47\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " Oct 31 15:12:39.268833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268665 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-socket\") pod \"be8f2787-9798-4c21-ae4e-7b1136b8b592\" (UID: \"be8f2787-9798-4c21-ae4e-7b1136b8b592\") " Oct 31 15:12:39.268833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268784 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-credential-socket\") pod \"a8392178-c8ba-4b53-9cec-fe944f3eac47\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " Oct 31 15:12:39.268976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268831 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-kube-api-access-r7g5z\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-kube-api-access-r7g5z\") pod \"a8392178-c8ba-4b53-9cec-fe944f3eac47\" (UID: \"a8392178-c8ba-4b53-9cec-fe944f3eac47\") " Oct 31 15:12:39.268976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.268800 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-data" pod "a8392178-c8ba-4b53-9cec-fe944f3eac47" (UID: "a8392178-c8ba-4b53-9cec-fe944f3eac47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.269069 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.269043 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-certs" pod "be8f2787-9798-4c21-ae4e-7b1136b8b592" (UID: "be8f2787-9798-4c21-ae4e-7b1136b8b592"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.269819 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.269335 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-socket" pod "be8f2787-9798-4c21-ae4e-7b1136b8b592" (UID: "be8f2787-9798-4c21-ae4e-7b1136b8b592"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.269819 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.269641 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-credential-socket" pod "be8f2787-9798-4c21-ae4e-7b1136b8b592" (UID: "be8f2787-9798-4c21-ae4e-7b1136b8b592"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.270879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.270796 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-data" pod "be8f2787-9798-4c21-ae4e-7b1136b8b592" (UID: "be8f2787-9798-4c21-ae4e-7b1136b8b592"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.271127 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.270980 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-token" pod "be8f2787-9798-4c21-ae4e-7b1136b8b592" (UID: "be8f2787-9798-4c21-ae4e-7b1136b8b592"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271551 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272450 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272479 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwmn8\" (UniqueName: \"kubernetes.io/projected/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-kube-api-access-hwmn8\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271183 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-certs" pod "a8392178-c8ba-4b53-9cec-fe944f3eac47" (UID: "a8392178-c8ba-4b53-9cec-fe944f3eac47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272496 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271347 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8392178-c8ba-4b53-9cec-fe944f3eac47-istiod-ca-cert" pod "a8392178-c8ba-4b53-9cec-fe944f3eac47" (UID: "a8392178-c8ba-4b53-9cec-fe944f3eac47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271418 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-podinfo" pod "a8392178-c8ba-4b53-9cec-fe944f3eac47" (UID: "a8392178-c8ba-4b53-9cec-fe944f3eac47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272516 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271425 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-podinfo" pod "be8f2787-9798-4c21-ae4e-7b1136b8b592" (UID: "be8f2787-9798-4c21-ae4e-7b1136b8b592"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271465 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-credential-socket" pod "a8392178-c8ba-4b53-9cec-fe944f3eac47" (UID: "a8392178-c8ba-4b53-9cec-fe944f3eac47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271485 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-envoy" pod "be8f2787-9798-4c21-ae4e-7b1136b8b592" (UID: "be8f2787-9798-4c21-ae4e-7b1136b8b592"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272540 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271544 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-kube-api-access-24x2d" pod "be8f2787-9798-4c21-ae4e-7b1136b8b592" (UID: "be8f2787-9798-4c21-ae4e-7b1136b8b592"). InnerVolumeSpecName "kube-api-access-24x2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272555 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d977f9c-25e6-451c-ae90-a0bbc31b2efd-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271608 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-envoy" pod "a8392178-c8ba-4b53-9cec-fe944f3eac47" (UID: "a8392178-c8ba-4b53-9cec-fe944f3eac47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.271785 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8f2787-9798-4c21-ae4e-7b1136b8b592-istiod-ca-cert" pod "be8f2787-9798-4c21-ae4e-7b1136b8b592" (UID: "be8f2787-9798-4c21-ae4e-7b1136b8b592"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272570 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272593 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272609 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272633 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272648 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272662 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272679 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272734 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.274138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.272749 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krggw\" (UniqueName: \"kubernetes.io/projected/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1-kube-api-access-krggw\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.275222 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.274495 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-token" pod "a8392178-c8ba-4b53-9cec-fe944f3eac47" (UID: "a8392178-c8ba-4b53-9cec-fe944f3eac47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.281310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.275578 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-kube-api-access-r7g5z" pod "a8392178-c8ba-4b53-9cec-fe944f3eac47" (UID: "a8392178-c8ba-4b53-9cec-fe944f3eac47"). InnerVolumeSpecName "kube-api-access-r7g5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:12:39.374497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374466 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374496 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374506 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374515 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374525 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7g5z\" (UniqueName: \"kubernetes.io/projected/a8392178-c8ba-4b53-9cec-fe944f3eac47-kube-api-access-r7g5z\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374536 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374545 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24x2d\" (UniqueName: \"kubernetes.io/projected/be8f2787-9798-4c21-ae4e-7b1136b8b592-kube-api-access-24x2d\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374554 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374564 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374572 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8392178-c8ba-4b53-9cec-fe944f3eac47-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374581 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374589 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8392178-c8ba-4b53-9cec-fe944f3eac47-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374598 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be8f2787-9798-4c21-ae4e-7b1136b8b592-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.374738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.374605 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be8f2787-9798-4c21-ae4e-7b1136b8b592-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:12:39.522588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.522488 4399 generic.go:356] "Generic (PLEG): container finished" podID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerID="8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170" exitCode=0 Oct 31 15:12:39.522588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.522546 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" Oct 31 15:12:39.522588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.522557 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" event={"ID":"be8f2787-9798-4c21-ae4e-7b1136b8b592","Type":"ContainerDied","Data":"8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170"} Oct 31 15:12:39.522929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.522670 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp" event={"ID":"be8f2787-9798-4c21-ae4e-7b1136b8b592","Type":"ContainerDied","Data":"00350befe1484ac797837fedff498841cc005d5ad3a900ebdc53be66e9d8cc81"} Oct 31 15:12:39.522929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.522725 4399 scope.go:122] "RemoveContainer" containerID="8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170" Oct 31 15:12:39.530730 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.530686 4399 generic.go:356] "Generic (PLEG): container finished" podID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerID="ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc" exitCode=0 Oct 31 15:12:39.530942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.530722 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" event={"ID":"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1","Type":"ContainerDied","Data":"ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc"} Oct 31 15:12:39.531064 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.530985 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" event={"ID":"5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1","Type":"ContainerDied","Data":"7b451dbbdc20c467501933f4d2de5df8837e69d8f1a5b01bd52650e737e18e99"} Oct 31 15:12:39.531064 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.530732 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb" Oct 31 15:12:39.534114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.533123 4399 generic.go:356] "Generic (PLEG): container finished" podID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerID="973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9" exitCode=0 Oct 31 15:12:39.534114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.533255 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" Oct 31 15:12:39.534520 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.534204 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" event={"ID":"2d977f9c-25e6-451c-ae90-a0bbc31b2efd","Type":"ContainerDied","Data":"973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9"} Oct 31 15:12:39.534520 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.534286 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d" event={"ID":"2d977f9c-25e6-451c-ae90-a0bbc31b2efd","Type":"ContainerDied","Data":"e9b77b61f7d71cd7b5b6a36c186eef3f94dc519fe497241f2f56cdcec14b950d"} Oct 31 15:12:39.536266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.536244 4399 generic.go:356] "Generic (PLEG): container finished" podID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerID="eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c" exitCode=0 Oct 31 15:12:39.536530 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.536478 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" event={"ID":"a8392178-c8ba-4b53-9cec-fe944f3eac47","Type":"ContainerDied","Data":"eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c"} Oct 31 15:12:39.536673 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.536655 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" event={"ID":"a8392178-c8ba-4b53-9cec-fe944f3eac47","Type":"ContainerDied","Data":"f66e839b5c5b3cfe16ce4a71605927151d432d01a209ce3798cd791df26d7f6e"} Oct 31 15:12:39.536865 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.536849 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9" Oct 31 15:12:39.542139 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.542121 4399 scope.go:122] "RemoveContainer" containerID="8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170" Oct 31 15:12:39.542446 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:39.542420 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170\": container with ID starting with 8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170 not found: ID does not exist" containerID="8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170" Oct 31 15:12:39.542849 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.542822 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170"} err="failed to get container status \"8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170\": rpc error: code = NotFound desc = could not find container \"8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170\": container with ID starting with 8f635cc2f43b390c1a983d6e4f94564abb0090716cfc35b524d334ef5ffb1170 not found: ID does not exist" Oct 31 15:12:39.542924 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.542852 4399 scope.go:122] "RemoveContainer" containerID="ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc" Oct 31 15:12:39.553997 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.553977 4399 scope.go:122] "RemoveContainer" containerID="ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc" Oct 31 15:12:39.554280 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:39.554258 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc\": container with ID starting with ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc not found: ID does not exist" containerID="ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc" Oct 31 15:12:39.554367 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.554287 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc"} err="failed to get container status \"ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc\": rpc error: code = NotFound desc = could not find container \"ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc\": container with ID starting with ee9c690b3996f548cfa22a0cb798e034d9fb73c39fdf7eb2fcd62ef1935549dc not found: ID does not exist" Oct 31 15:12:39.554367 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.554304 4399 scope.go:122] "RemoveContainer" containerID="973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9" Oct 31 15:12:39.563947 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.563928 4399 scope.go:122] "RemoveContainer" containerID="973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9" Oct 31 15:12:39.564231 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:39.564213 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9\": container with ID starting with 973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9 not found: ID does not exist" containerID="973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9" Oct 31 15:12:39.564314 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.564241 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9"} err="failed to get container status \"973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9\": rpc error: code = NotFound desc = could not find container \"973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9\": container with ID starting with 973462aa4255757cc890d2ecde9ba430bf6a9f8b6388e13846df2dda3aa036b9 not found: ID does not exist" Oct 31 15:12:39.564314 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.564265 4399 scope.go:122] "RemoveContainer" containerID="eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c" Oct 31 15:12:39.573017 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.572996 4399 scope.go:122] "RemoveContainer" containerID="eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c" Oct 31 15:12:39.573269 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:39.573251 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c\": container with ID starting with eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c not found: ID does not exist" containerID="eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c" Oct 31 15:12:39.573329 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.573276 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c"} err="failed to get container status \"eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c\": rpc error: code = NotFound desc = could not find container \"eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c\": container with ID starting with eade14ef14fac579dd9c0f8555206a10ade413f2823a296ef565996a32547a3c not found: ID does not exist" Oct 31 15:12:39.716890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.716859 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb"] Oct 31 15:12:39.720148 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.720117 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-4gtcb"] Oct 31 15:12:39.821864 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.821773 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9"] Oct 31 15:12:39.826110 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.826086 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-r45d9"] Oct 31 15:12:39.931532 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.931428 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d"] Oct 31 15:12:39.934649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:39.934627 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-vdr6d"] Oct 31 15:12:40.045656 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:40.045630 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp"] Oct 31 15:12:40.048657 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:40.048638 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-lb4xp"] Oct 31 15:12:40.898571 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:40.898539 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" path="/var/lib/kubelet/pods/2d977f9c-25e6-451c-ae90-a0bbc31b2efd/volumes" Oct 31 15:12:40.899142 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:40.899125 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" path="/var/lib/kubelet/pods/5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1/volumes" Oct 31 15:12:40.899556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:40.899543 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" path="/var/lib/kubelet/pods/a8392178-c8ba-4b53-9cec-fe944f3eac47/volumes" Oct 31 15:12:40.900403 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:40.900382 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" path="/var/lib/kubelet/pods/be8f2787-9798-4c21-ae4e-7b1136b8b592/volumes" Oct 31 15:12:43.481548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:43.481515 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:12:44.488035 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:44.488004 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:12:46.972508 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.972471 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d"] Oct 31 15:12:46.973197 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973178 4399 cpu_manager.go:414] "containerMap: removing container" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" Oct 31 15:12:46.973197 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973198 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" Oct 31 15:12:46.973282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973216 4399 cpu_manager.go:414] "containerMap: removing container" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" Oct 31 15:12:46.973282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973223 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" Oct 31 15:12:46.973282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973241 4399 cpu_manager.go:414] "containerMap: removing container" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" Oct 31 15:12:46.973282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973245 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" Oct 31 15:12:46.973282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973267 4399 cpu_manager.go:414] "containerMap: removing container" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" Oct 31 15:12:46.973282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973272 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" Oct 31 15:12:46.973470 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973358 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a3f9bdf-54b9-44aa-9d6d-8d6d0f2366d1" containerName="istio-proxy" Oct 31 15:12:46.973470 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973371 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8392178-c8ba-4b53-9cec-fe944f3eac47" containerName="istio-proxy" Oct 31 15:12:46.973470 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973379 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d977f9c-25e6-451c-ae90-a0bbc31b2efd" containerName="istio-proxy" Oct 31 15:12:46.973470 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.973387 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8f2787-9798-4c21-ae4e-7b1136b8b592" containerName="istio-proxy" Oct 31 15:12:46.976806 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:46.976789 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.021115 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.021083 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d"] Oct 31 15:12:47.032839 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.032810 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7bh\" (UniqueName: \"kubernetes.io/projected/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-kube-api-access-7k7bh\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.032983 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.032877 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.033092 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.033067 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-metrics-certs\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.033199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.033179 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.073921 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.067524 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f7251e-serving-cert\"" Oct 31 15:12:47.134038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.134011 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.134196 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.134100 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7bh\" (UniqueName: \"kubernetes.io/projected/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-kube-api-access-7k7bh\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.134196 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.134183 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.134387 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:47.134371 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-serving-cert: secret "maas-api-e2e-api-b-f7251e-serving-cert" not found Oct 31 15:12:47.134553 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.134534 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-metrics-certs\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.134669 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:47.134654 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls podName:db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e nodeName:}" failed. No retries permitted until 2026-10-31 15:12:47.634636049 +0000 UTC m=+1610.845877443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls") pod "maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" (UID: "db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e") : secret "maas-api-e2e-api-b-f7251e-serving-cert" not found Oct 31 15:12:47.134836 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.134814 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-openshift-service-ca\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.137134 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.137110 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-metrics-certs\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.191532 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.191453 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7bh\" (UniqueName: \"kubernetes.io/projected/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-kube-api-access-7k7bh\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.655726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.645762 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.671690 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.671647 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls\") pod \"maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:47.890398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:47.890371 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:48.277360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:48.277278 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d"] Oct 31 15:12:48.281683 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:12:48.281650 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0f2ad7_3c56_4fb4_81bb_5a7bbbe9f73e.slice/crio-60aed8478ed133e618859baad0e6be3177b6f9f684680c4f4a29d3fa2519e82d WatchSource:0}: Error finding container 60aed8478ed133e618859baad0e6be3177b6f9f684680c4f4a29d3fa2519e82d: Status 404 returned error can't find the container with id 60aed8478ed133e618859baad0e6be3177b6f9f684680c4f4a29d3fa2519e82d Oct 31 15:12:48.624464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:48.624377 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" event={"ID":"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e","Type":"ContainerStarted","Data":"60aed8478ed133e618859baad0e6be3177b6f9f684680c4f4a29d3fa2519e82d"} Oct 31 15:12:49.634715 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:49.634664 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" event={"ID":"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e","Type":"ContainerStarted","Data":"ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9"} Oct 31 15:12:49.635172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:49.634905 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:49.750301 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:49.750244 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" podStartSLOduration=2.920160254 podStartE2EDuration="3.750229643s" totalImagesPullingTime="830.069389ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:12:49.74705276 +0000 UTC m=+1612.958294173" watchObservedRunningTime="2026-10-31 15:12:49.750229643 +0000 UTC m=+1612.961471055" Oct 31 15:12:55.647770 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:12:55.647734 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:12:56.208080 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:12:56.208046 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:13:03.170400 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.170371 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht"] Oct 31 15:13:03.176631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.176610 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.229293 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.229270 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2196e4-serving-cert\"" Oct 31 15:13:03.282147 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.282117 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht"] Oct 31 15:13:03.287752 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.287730 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-metrics-certs\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.287868 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.287788 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.287868 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.287855 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlw6c\" (UniqueName: \"kubernetes.io/projected/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-kube-api-access-vlw6c\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.287942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.287897 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.389472 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.389439 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-metrics-certs\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.389632 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.389495 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.389632 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.389520 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlw6c\" (UniqueName: \"kubernetes.io/projected/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-kube-api-access-vlw6c\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.389632 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.389549 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.389787 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:13:03.389651 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-serving-cert: secret "maas-api-e2e-ipp-a-2196e4-serving-cert" not found Oct 31 15:13:03.389787 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:13:03.389772 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls podName:e2a4c047-3ed2-4ad3-89b1-6349b35fbad5 nodeName:}" failed. No retries permitted until 2026-10-31 15:13:03.88975111 +0000 UTC m=+1627.100992520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls") pod "maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" (UID: "e2a4c047-3ed2-4ad3-89b1-6349b35fbad5") : secret "maas-api-e2e-ipp-a-2196e4-serving-cert" not found Oct 31 15:13:03.390308 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.390285 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.392001 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.391983 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-metrics-certs\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.447179 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.447150 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlw6c\" (UniqueName: \"kubernetes.io/projected/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-kube-api-access-vlw6c\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.896690 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.896607 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:03.899013 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:03.898989 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:04.090387 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:04.090356 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:04.470073 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:04.470041 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht"] Oct 31 15:13:04.471313 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:13:04.471285 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a4c047_3ed2_4ad3_89b1_6349b35fbad5.slice/crio-d3c65c569de23d9c811f5286cded3d8ff0726e94f510ed4ed309798c01416550 WatchSource:0}: Error finding container d3c65c569de23d9c811f5286cded3d8ff0726e94f510ed4ed309798c01416550: Status 404 returned error can't find the container with id d3c65c569de23d9c811f5286cded3d8ff0726e94f510ed4ed309798c01416550 Oct 31 15:13:04.771198 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:04.771119 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" event={"ID":"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5","Type":"ContainerStarted","Data":"d3c65c569de23d9c811f5286cded3d8ff0726e94f510ed4ed309798c01416550"} Oct 31 15:13:05.779955 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:05.779923 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" event={"ID":"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5","Type":"ContainerStarted","Data":"a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40"} Oct 31 15:13:05.780340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:05.780012 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:05.897396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:05.897350 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" podStartSLOduration=1.9599506180000001 podStartE2EDuration="2.897336432s" totalImagesPullingTime="937.385814ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:05.894881047 +0000 UTC m=+1629.106122478" watchObservedRunningTime="2026-10-31 15:13:05.897336432 +0000 UTC m=+1629.108577845" Oct 31 15:13:11.793253 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:11.793223 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:13:13.660506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.660472 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df"] Oct 31 15:13:13.665325 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.665307 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:13.720457 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.720434 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df"] Oct 31 15:13:13.766328 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.766300 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-644a8a-dockercfg-j6pzf\"" Oct 31 15:13:13.766462 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.766422 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-644a8a\"" Oct 31 15:13:13.838305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.838275 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a616bb05-16fb-40e1-a4f7-7c5a396662f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df\" (UID: \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:13.838444 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.838424 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dr57\" (UniqueName: \"kubernetes.io/projected/a616bb05-16fb-40e1-a4f7-7c5a396662f5-kube-api-access-4dr57\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df\" (UID: \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:13.940001 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.939923 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a616bb05-16fb-40e1-a4f7-7c5a396662f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df\" (UID: \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:13.940151 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.940020 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dr57\" (UniqueName: \"kubernetes.io/projected/a616bb05-16fb-40e1-a4f7-7c5a396662f5-kube-api-access-4dr57\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df\" (UID: \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:13.940911 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.940878 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a616bb05-16fb-40e1-a4f7-7c5a396662f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df\" (UID: \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:13.997345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:13.997320 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dr57\" (UniqueName: \"kubernetes.io/projected/a616bb05-16fb-40e1-a4f7-7c5a396662f5-kube-api-access-4dr57\") pod \"payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df\" (UID: \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:14.279496 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:14.279466 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:14.658899 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:14.658874 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df"] Oct 31 15:13:14.659787 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:13:14.659762 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda616bb05_16fb_40e1_a4f7_7c5a396662f5.slice/crio-ad8a1d72fc28082ecf14da73d544e2bcadfc8e464af72b152c6fdcccbf8eeb90 WatchSource:0}: Error finding container ad8a1d72fc28082ecf14da73d544e2bcadfc8e464af72b152c6fdcccbf8eeb90: Status 404 returned error can't find the container with id ad8a1d72fc28082ecf14da73d544e2bcadfc8e464af72b152c6fdcccbf8eeb90 Oct 31 15:13:14.844769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:14.844734 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" event={"ID":"a616bb05-16fb-40e1-a4f7-7c5a396662f5","Type":"ContainerStarted","Data":"ad8a1d72fc28082ecf14da73d544e2bcadfc8e464af72b152c6fdcccbf8eeb90"} Oct 31 15:13:16.860254 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:16.860208 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" event={"ID":"a616bb05-16fb-40e1-a4f7-7c5a396662f5","Type":"ContainerStarted","Data":"220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b"} Oct 31 15:13:16.860797 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:16.860368 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:16.975380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:16.975323 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" podStartSLOduration=2.274447963 podStartE2EDuration="3.975308495s" totalImagesPullingTime="1.700860532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:16.973412021 +0000 UTC m=+1640.184653435" watchObservedRunningTime="2026-10-31 15:13:16.975308495 +0000 UTC m=+1640.186549907" Oct 31 15:13:19.758844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.758811 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2"] Oct 31 15:13:19.763160 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.763143 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.864948 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.864908 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7f41-b0db-48f9-bc23-153d28d62304-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.865237 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.865207 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.865390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.865277 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.865458 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.865414 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.865458 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.865443 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.865564 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.865481 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.865728 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.865683 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.865853 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.865817 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.865918 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.865867 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9fc\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-kube-api-access-9s9fc\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.869211 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.869187 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2"] Oct 31 15:13:19.967735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.967486 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.967735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.967548 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9fc\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-kube-api-access-9s9fc\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.967735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.967633 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7f41-b0db-48f9-bc23-153d28d62304-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.968534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.968387 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.968534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.968453 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.969062 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.968824 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.969062 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.968876 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.969062 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.968985 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.969361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.969334 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.971141 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.970525 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.971498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.971447 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.972834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.972466 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7f41-b0db-48f9-bc23-153d28d62304-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.972834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.972795 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.996200 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.996167 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:19.998145 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:19.998121 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:20.014331 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.014251 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:20.029606 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.029568 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:20.030173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.030150 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9fc\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-kube-api-access-9s9fc\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:20.077218 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.077197 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:20.507731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.507678 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2"] Oct 31 15:13:20.508751 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:13:20.508682 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba7a7f41_b0db_48f9_bc23_153d28d62304.slice/crio-52b0077fdd5a3008ae3439e1a52e1652f8ff7b168bf3cc372246a1315497e542 WatchSource:0}: Error finding container 52b0077fdd5a3008ae3439e1a52e1652f8ff7b168bf3cc372246a1315497e542: Status 404 returned error can't find the container with id 52b0077fdd5a3008ae3439e1a52e1652f8ff7b168bf3cc372246a1315497e542 Oct 31 15:13:20.511541 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.511508 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:13:20.511622 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.511577 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:13:20.511622 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.511604 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:13:20.896598 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.896523 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" event={"ID":"ba7a7f41-b0db-48f9-bc23-153d28d62304","Type":"ContainerStarted","Data":"90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5"} Oct 31 15:13:20.896598 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:20.896572 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" event={"ID":"ba7a7f41-b0db-48f9-bc23-153d28d62304","Type":"ContainerStarted","Data":"52b0077fdd5a3008ae3439e1a52e1652f8ff7b168bf3cc372246a1315497e542"} Oct 31 15:13:21.008873 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:21.008812 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podStartSLOduration=2.008794138 podStartE2EDuration="2.008794138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:21.005149398 +0000 UTC m=+1644.216390823" watchObservedRunningTime="2026-10-31 15:13:21.008794138 +0000 UTC m=+1644.220035551" Oct 31 15:13:21.078346 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:21.078315 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:21.080420 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:21.080395 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 15:13:21.080559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:21.080436 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 15:13:22.077893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:22.077856 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 15:13:22.078286 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:22.077936 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 15:13:23.078342 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:23.078307 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 15:13:23.078757 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:23.078356 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 15:13:24.077861 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:24.077829 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 15:13:24.078044 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:24.077891 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 15:13:24.900903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:24.900873 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn"] Oct 31 15:13:24.905676 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:24.905653 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:24.972277 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:24.972240 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn"] Oct 31 15:13:25.011537 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.011507 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-94a889-serving-cert\"" Oct 31 15:13:25.032260 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.032228 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.032425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.032316 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da85bae-56c8-429c-a5df-e48336d6bb7e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.032425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.032366 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-metrics-certs\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.032566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.032439 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xg8b\" (UniqueName: \"kubernetes.io/projected/8da85bae-56c8-429c-a5df-e48336d6bb7e-kube-api-access-2xg8b\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.077859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.077829 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 15:13:25.078044 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.077895 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 15:13:25.134306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.133478 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.134306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.133609 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da85bae-56c8-429c-a5df-e48336d6bb7e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.134306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.133677 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-metrics-certs\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.134306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.133887 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg8b\" (UniqueName: \"kubernetes.io/projected/8da85bae-56c8-429c-a5df-e48336d6bb7e-kube-api-access-2xg8b\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.135916 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.135891 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da85bae-56c8-429c-a5df-e48336d6bb7e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.150080 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.150046 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.175406 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.175328 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-metrics-certs\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.193001 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.192966 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xg8b\" (UniqueName: \"kubernetes.io/projected/8da85bae-56c8-429c-a5df-e48336d6bb7e-kube-api-access-2xg8b\") pod \"maas-api-e2e-ipp-b-94a889-5798548757-xxxcn\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.224037 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.224002 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:25.634092 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.634058 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn"] Oct 31 15:13:25.635230 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:13:25.635205 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da85bae_56c8_429c_a5df_e48336d6bb7e.slice/crio-d8332745835440ad002bce9a57a54dae262825cb72271c6edd7568cc75d8d945 WatchSource:0}: Error finding container d8332745835440ad002bce9a57a54dae262825cb72271c6edd7568cc75d8d945: Status 404 returned error can't find the container with id d8332745835440ad002bce9a57a54dae262825cb72271c6edd7568cc75d8d945 Oct 31 15:13:25.926525 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:25.926445 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" event={"ID":"8da85bae-56c8-429c-a5df-e48336d6bb7e","Type":"ContainerStarted","Data":"d8332745835440ad002bce9a57a54dae262825cb72271c6edd7568cc75d8d945"} Oct 31 15:13:26.078149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:26.078110 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 15:13:26.078296 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:26.078160 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 15:13:26.208351 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:13:26.208312 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:13:26.934146 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:26.934066 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" event={"ID":"8da85bae-56c8-429c-a5df-e48336d6bb7e","Type":"ContainerStarted","Data":"a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724"} Oct 31 15:13:26.934505 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:26.934258 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:27.050389 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:27.050329 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" podStartSLOduration=2.173783268 podStartE2EDuration="3.050305745s" totalImagesPullingTime="876.522477ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:27.047027516 +0000 UTC m=+1650.258268941" watchObservedRunningTime="2026-10-31 15:13:27.050305745 +0000 UTC m=+1650.261547156" Oct 31 15:13:27.078450 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:27.078415 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 31 15:13:27.078790 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:27.078479 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 31 15:13:27.869949 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:27.869912 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:13:28.083281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:28.083249 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:28.083752 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:28.083735 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:28.084369 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:28.084353 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:13:32.946151 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:32.946124 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:13:34.883526 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.883496 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg"] Oct 31 15:13:34.887806 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.887784 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.891483 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.891455 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.891610 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.891592 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9e388b-1fab-43da-8991-1fe0df782480-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.891676 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.891661 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.891747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.891709 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.891809 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.891794 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.891863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.891847 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvsc7\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-kube-api-access-tvsc7\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.891946 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.891928 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.892011 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.891977 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9e388b-1fab-43da-8991-1fe0df782480-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.892086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.892065 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.933380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.933354 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz"] Oct 31 15:13:34.937672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.937652 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.989333 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.989307 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg"] Oct 31 15:13:34.993073 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.993045 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.993221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.993083 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.993384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.993351 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.993510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.993431 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvsc7\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-kube-api-access-tvsc7\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.993510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.993460 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxfx2\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-kube-api-access-lxfx2\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.993510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.993490 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.993678 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.993488 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.995272 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995244 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48578e0d-a009-4589-bf9a-54f4acdd7416-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.995382 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995281 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9e388b-1fab-43da-8991-1fe0df782480-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.995382 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995365 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.995505 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995417 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.995505 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995465 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.995614 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995526 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.995670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995627 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.995747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995672 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.995811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995748 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:34.995870 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995806 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9e388b-1fab-43da-8991-1fe0df782480-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.995870 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995828 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.995973 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995939 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.996025 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.995975 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.996131 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.996056 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9e388b-1fab-43da-8991-1fe0df782480-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.996404 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.996385 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.996772 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.996733 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.998974 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.998948 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:34.999070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:34.999055 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9e388b-1fab-43da-8991-1fe0df782480-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:35.061460 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.039824 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz"] Oct 31 15:13:35.061460 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.055063 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:35.061460 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.055488 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvsc7\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-kube-api-access-tvsc7\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:35.097406 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.097375 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.097569 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.097526 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.097569 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.097565 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.097719 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.097613 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxfx2\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-kube-api-access-lxfx2\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.097719 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.097650 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48578e0d-a009-4589-bf9a-54f4acdd7416-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.097829 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.097755 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.097883 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.097831 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.097942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.097933 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.098001 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.097969 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.099652 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.099587 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-certs\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.099825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.099749 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48578e0d-a009-4589-bf9a-54f4acdd7416-istiod-ca-cert\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.099910 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.099884 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-data\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.100389 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.100300 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-credential-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.100736 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.100672 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-socket\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.103430 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.103410 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-podinfo\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.106956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.106933 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-envoy\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.177475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.175750 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-token\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.177475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.176086 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxfx2\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-kube-api-access-lxfx2\") pod \"e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.202137 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.202108 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:35.252431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.252398 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:35.640521 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.640481 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg"] Oct 31 15:13:35.681107 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:13:35.681078 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9e388b_1fab_43da_8991_1fe0df782480.slice/crio-3d0305d90a9e9a5655e37eb87f0c031654885a881c66271522bdb7c925648f68 WatchSource:0}: Error finding container 3d0305d90a9e9a5655e37eb87f0c031654885a881c66271522bdb7c925648f68: Status 404 returned error can't find the container with id 3d0305d90a9e9a5655e37eb87f0c031654885a881c66271522bdb7c925648f68 Oct 31 15:13:35.684445 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.684409 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:13:35.684539 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.684482 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:13:35.684539 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.684511 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:13:35.694149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.694127 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz"] Oct 31 15:13:35.695829 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:13:35.695808 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48578e0d_a009_4589_bf9a_54f4acdd7416.slice/crio-8aef888d300a11274377e8ed69245f1815cff998ff022babe71ddd276d4d78e2 WatchSource:0}: Error finding container 8aef888d300a11274377e8ed69245f1815cff998ff022babe71ddd276d4d78e2: Status 404 returned error can't find the container with id 8aef888d300a11274377e8ed69245f1815cff998ff022babe71ddd276d4d78e2 Oct 31 15:13:35.698341 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.698312 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:13:35.698434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.698378 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:13:35.698434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.698416 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:13:35.999724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.999395 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" event={"ID":"2c9e388b-1fab-43da-8991-1fe0df782480","Type":"ContainerStarted","Data":"8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5"} Oct 31 15:13:35.999724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:35.999454 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" event={"ID":"2c9e388b-1fab-43da-8991-1fe0df782480","Type":"ContainerStarted","Data":"3d0305d90a9e9a5655e37eb87f0c031654885a881c66271522bdb7c925648f68"} Oct 31 15:13:36.004746 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.001758 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" event={"ID":"48578e0d-a009-4589-bf9a-54f4acdd7416","Type":"ContainerStarted","Data":"18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700"} Oct 31 15:13:36.004746 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.001816 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" event={"ID":"48578e0d-a009-4589-bf9a-54f4acdd7416","Type":"ContainerStarted","Data":"8aef888d300a11274377e8ed69245f1815cff998ff022babe71ddd276d4d78e2"} Oct 31 15:13:36.115786 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.115737 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podStartSLOduration=2.115718005 podStartE2EDuration="2.115718005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:36.114116134 +0000 UTC m=+1659.325357550" watchObservedRunningTime="2026-10-31 15:13:36.115718005 +0000 UTC m=+1659.326959417" Oct 31 15:13:36.203199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.203073 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:36.204583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.204550 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 15:13:36.204772 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.204612 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 15:13:36.235723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.235634 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" podStartSLOduration=2.235615309 podStartE2EDuration="2.235615309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:13:36.229590409 +0000 UTC m=+1659.440831824" watchObservedRunningTime="2026-10-31 15:13:36.235615309 +0000 UTC m=+1659.446856722" Oct 31 15:13:36.253319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.253230 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:36.257443 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.255855 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 15:13:36.257443 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:36.255918 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 15:13:37.202442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:37.202412 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 15:13:37.202866 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:37.202472 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 15:13:37.254104 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:37.253818 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 15:13:37.254104 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:37.253879 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 15:13:38.203509 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:38.203473 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 15:13:38.203922 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:38.203531 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 15:13:38.264149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:38.264112 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 15:13:38.264442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:38.264410 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 15:13:39.203340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:39.203299 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 15:13:39.203524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:39.203376 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 15:13:39.254723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:39.254347 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 15:13:39.254723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:39.254406 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 15:13:40.202865 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:40.202821 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 15:13:40.203070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:40.202890 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 15:13:40.253170 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:40.253137 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 15:13:40.253425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:40.253394 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 15:13:41.203265 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:41.203231 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 15:13:41.203686 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:41.203297 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 15:13:41.254744 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:41.254163 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body= Oct 31 15:13:41.254744 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:41.254219 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" Oct 31 15:13:42.203370 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:42.203336 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 31 15:13:42.203798 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:42.203389 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 31 15:13:42.424920 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:42.424886 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:42.425688 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:42.425661 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:42.426530 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:42.426510 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:13:43.207200 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:43.207172 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:43.207616 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:43.207564 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:43.208263 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:43.208241 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:13:56.208276 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:13:56.208238 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:13:58.289491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:58.289455 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm"] Oct 31 15:13:58.289962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:58.289654 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" containerID="cri-o://762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9" gracePeriod=30 Oct 31 15:13:58.305153 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:58.305119 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb"] Oct 31 15:13:58.305335 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:58.305315 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" containerID="cri-o://284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703" gracePeriod=30 Oct 31 15:13:59.614670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:59.614642 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:13:59.615047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:59.614684 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:13:59.858249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:59.858219 4399 patch_prober.go:29] interesting pod/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:13:59.858430 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:13:59.858272 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:00.495161 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.495125 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft"] Oct 31 15:14:00.500824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.500801 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.573750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.573685 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea262732-121b-484c-bcc7-cfe929b2147f-istio-podinfo\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.573905 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.573780 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-certs\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.574011 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.573984 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-istio-token\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.574105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.574087 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-credential-socket\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.574181 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.574166 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnqgk\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-kube-api-access-nnqgk\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.574213 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.574205 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea262732-121b-484c-bcc7-cfe929b2147f-istiod-ca-cert\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.574289 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.574279 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-envoy\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.574365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.574354 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-socket\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.574399 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.574373 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-data\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.601213 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.601184 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft"] Oct 31 15:14:00.646012 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.645966 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-032b05-openshift-default-dockercfg-xvtds\"" Oct 31 15:14:00.676043 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676017 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnqgk\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-kube-api-access-nnqgk\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676047 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea262732-121b-484c-bcc7-cfe929b2147f-istiod-ca-cert\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676105 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-envoy\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676166 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-socket\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676192 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-data\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676230 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea262732-121b-484c-bcc7-cfe929b2147f-istio-podinfo\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676269 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-certs\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676551 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676412 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-istio-token\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676551 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676481 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-credential-socket\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676676 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676616 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-socket\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676728 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-data\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676853 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676827 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-certs\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676931 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676898 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-credential-socket\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.676986 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.676930 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea262732-121b-484c-bcc7-cfe929b2147f-istiod-ca-cert\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.678785 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.678764 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-envoy\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.678887 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.678869 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea262732-121b-484c-bcc7-cfe929b2147f-istio-podinfo\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.732249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.732224 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnqgk\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-kube-api-access-nnqgk\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.732767 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.732745 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-istio-token\") pod \"e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:00.815655 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:00.815567 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:01.256444 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:01.256421 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft"] Oct 31 15:14:01.257159 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:14:01.257132 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea262732_121b_484c_bcc7_cfe929b2147f.slice/crio-d0dd306d9e4768b82fe3e9e2a53d75ac4e3def81415e29bf89d7457ee61477fa WatchSource:0}: Error finding container d0dd306d9e4768b82fe3e9e2a53d75ac4e3def81415e29bf89d7457ee61477fa: Status 404 returned error can't find the container with id d0dd306d9e4768b82fe3e9e2a53d75ac4e3def81415e29bf89d7457ee61477fa Oct 31 15:14:01.259752 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:01.259721 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:14:01.259835 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:01.259787 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:14:01.259835 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:01.259824 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:14:02.187130 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:02.187089 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" event={"ID":"ea262732-121b-484c-bcc7-cfe929b2147f","Type":"ContainerStarted","Data":"25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51"} Oct 31 15:14:02.187606 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:02.187156 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" event={"ID":"ea262732-121b-484c-bcc7-cfe929b2147f","Type":"ContainerStarted","Data":"d0dd306d9e4768b82fe3e9e2a53d75ac4e3def81415e29bf89d7457ee61477fa"} Oct 31 15:14:02.356378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:02.356326 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" podStartSLOduration=2.356305253 podStartE2EDuration="2.356305253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:14:02.352674234 +0000 UTC m=+1685.563915650" watchObservedRunningTime="2026-10-31 15:14:02.356305253 +0000 UTC m=+1685.567546729" Oct 31 15:14:02.816244 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:02.816214 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:02.822115 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:02.822088 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:03.193778 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.193661 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:03.194594 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.194576 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:14:03.611496 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.611465 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:14:03.657344 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.657320 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:14:03.725562 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.725529 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-podinfo\") pod \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " Oct 31 15:14:03.725761 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.725612 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-credential-socket\") pod \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " Oct 31 15:14:03.725761 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.725666 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-certs\") pod \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " Oct 31 15:14:03.725761 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.725751 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-socket\") pod \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " Oct 31 15:14:03.725914 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.725792 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-credential-socket\") pod \"954d2d01-a14b-40b4-8f26-469bf2909f91\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " Oct 31 15:14:03.725914 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.725827 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-certs\") pod \"954d2d01-a14b-40b4-8f26-469bf2909f91\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " Oct 31 15:14:03.726021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.725927 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-envoy\") pod \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " Oct 31 15:14:03.726021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.725977 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-envoy\") pod \"954d2d01-a14b-40b4-8f26-469bf2909f91\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " Oct 31 15:14:03.726124 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726033 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-token\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-token\") pod \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " Oct 31 15:14:03.726124 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726075 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-socket\") pod \"954d2d01-a14b-40b4-8f26-469bf2909f91\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " Oct 31 15:14:03.726124 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726002 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-credential-socket" pod "b9461a39-85f3-43c8-a510-d8d8a184e9fa" (UID: "b9461a39-85f3-43c8-a510-d8d8a184e9fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.726281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726047 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-certs" pod "b9461a39-85f3-43c8-a510-d8d8a184e9fa" (UID: "b9461a39-85f3-43c8-a510-d8d8a184e9fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.726281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726053 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-credential-socket" pod "954d2d01-a14b-40b4-8f26-469bf2909f91" (UID: "954d2d01-a14b-40b4-8f26-469bf2909f91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.726281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726097 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-socket" pod "b9461a39-85f3-43c8-a510-d8d8a184e9fa" (UID: "b9461a39-85f3-43c8-a510-d8d8a184e9fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.726281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726114 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-kube-api-access-wkftw\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-kube-api-access-wkftw\") pod \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " Oct 31 15:14:03.726281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726250 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-token\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-token\") pod \"954d2d01-a14b-40b4-8f26-469bf2909f91\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " Oct 31 15:14:03.726615 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726292 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istiod-ca-cert\") pod \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " Oct 31 15:14:03.726615 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726324 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-kube-api-access-rv87c\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-kube-api-access-rv87c\") pod \"954d2d01-a14b-40b4-8f26-469bf2909f91\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " Oct 31 15:14:03.726615 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726423 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-socket" pod "954d2d01-a14b-40b4-8f26-469bf2909f91" (UID: "954d2d01-a14b-40b4-8f26-469bf2909f91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.726615 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726541 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-data\") pod \"954d2d01-a14b-40b4-8f26-469bf2909f91\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " Oct 31 15:14:03.726615 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726590 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-podinfo\") pod \"954d2d01-a14b-40b4-8f26-469bf2909f91\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " Oct 31 15:14:03.726944 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.726617 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954d2d01-a14b-40b4-8f26-469bf2909f91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954d2d01-a14b-40b4-8f26-469bf2909f91-istiod-ca-cert\") pod \"954d2d01-a14b-40b4-8f26-469bf2909f91\" (UID: \"954d2d01-a14b-40b4-8f26-469bf2909f91\") " Oct 31 15:14:03.727409 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.727102 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-certs" pod "954d2d01-a14b-40b4-8f26-469bf2909f91" (UID: "954d2d01-a14b-40b4-8f26-469bf2909f91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.728337 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.728219 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-data\") pod \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\" (UID: \"b9461a39-85f3-43c8-a510-d8d8a184e9fa\") " Oct 31 15:14:03.730197 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.730170 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.730284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.730204 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.730284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.730220 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.730284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.730237 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.730452 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.730250 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.730452 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.730299 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-data" pod "954d2d01-a14b-40b4-8f26-469bf2909f91" (UID: "954d2d01-a14b-40b4-8f26-469bf2909f91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.730547 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.730472 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-data" pod "b9461a39-85f3-43c8-a510-d8d8a184e9fa" (UID: "b9461a39-85f3-43c8-a510-d8d8a184e9fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.731485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.731436 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954d2d01-a14b-40b4-8f26-469bf2909f91-istiod-ca-cert" pod "954d2d01-a14b-40b4-8f26-469bf2909f91" (UID: "954d2d01-a14b-40b4-8f26-469bf2909f91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:03.731803 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.731772 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istiod-ca-cert" pod "b9461a39-85f3-43c8-a510-d8d8a184e9fa" (UID: "b9461a39-85f3-43c8-a510-d8d8a184e9fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:03.731879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.731834 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-podinfo" pod "b9461a39-85f3-43c8-a510-d8d8a184e9fa" (UID: "b9461a39-85f3-43c8-a510-d8d8a184e9fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:03.732295 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.732065 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-envoy" pod "b9461a39-85f3-43c8-a510-d8d8a184e9fa" (UID: "b9461a39-85f3-43c8-a510-d8d8a184e9fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.732295 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.732087 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-kube-api-access-wkftw" pod "b9461a39-85f3-43c8-a510-d8d8a184e9fa" (UID: "b9461a39-85f3-43c8-a510-d8d8a184e9fa"). InnerVolumeSpecName "kube-api-access-wkftw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.732295 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.732121 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-kube-api-access-rv87c" pod "954d2d01-a14b-40b4-8f26-469bf2909f91" (UID: "954d2d01-a14b-40b4-8f26-469bf2909f91"). InnerVolumeSpecName "kube-api-access-rv87c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.732542 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.732444 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-token" pod "b9461a39-85f3-43c8-a510-d8d8a184e9fa" (UID: "b9461a39-85f3-43c8-a510-d8d8a184e9fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.733152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.733130 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-token" pod "954d2d01-a14b-40b4-8f26-469bf2909f91" (UID: "954d2d01-a14b-40b4-8f26-469bf2909f91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:03.733152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.733137 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-envoy" pod "954d2d01-a14b-40b4-8f26-469bf2909f91" (UID: "954d2d01-a14b-40b4-8f26-469bf2909f91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:03.733632 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.733603 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-podinfo" pod "954d2d01-a14b-40b4-8f26-469bf2909f91" (UID: "954d2d01-a14b-40b4-8f26-469bf2909f91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:03.831647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831534 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.831647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831568 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkftw\" (UniqueName: \"kubernetes.io/projected/b9461a39-85f3-43c8-a510-d8d8a184e9fa-kube-api-access-wkftw\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.831647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831583 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.831647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831597 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.831647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831609 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv87c\" (UniqueName: \"kubernetes.io/projected/954d2d01-a14b-40b4-8f26-469bf2909f91-kube-api-access-rv87c\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.831647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831621 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.831647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831633 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.831647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831646 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954d2d01-a14b-40b4-8f26-469bf2909f91-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.832070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831659 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.832070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831671 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.832070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831709 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.832070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831723 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9461a39-85f3-43c8-a510-d8d8a184e9fa-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:03.832070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:03.831736 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954d2d01-a14b-40b4-8f26-469bf2909f91-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:04.201499 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.201464 4399 generic.go:356] "Generic (PLEG): container finished" podID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerID="762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9" exitCode=0 Oct 31 15:14:04.202008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.201508 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" Oct 31 15:14:04.202008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.201552 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" event={"ID":"954d2d01-a14b-40b4-8f26-469bf2909f91","Type":"ContainerDied","Data":"762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9"} Oct 31 15:14:04.202008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.201608 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm" event={"ID":"954d2d01-a14b-40b4-8f26-469bf2909f91","Type":"ContainerDied","Data":"6567a3a9b3b00ec3da0c11ab93daec4b62ac39f150847b5f241295743e67d1bb"} Oct 31 15:14:04.202008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.201636 4399 scope.go:122] "RemoveContainer" containerID="762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9" Oct 31 15:14:04.203362 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.203339 4399 generic.go:356] "Generic (PLEG): container finished" podID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerID="284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703" exitCode=0 Oct 31 15:14:04.203506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.203485 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" event={"ID":"b9461a39-85f3-43c8-a510-d8d8a184e9fa","Type":"ContainerDied","Data":"284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703"} Oct 31 15:14:04.203560 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.203533 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" event={"ID":"b9461a39-85f3-43c8-a510-d8d8a184e9fa","Type":"ContainerDied","Data":"ecfbe7ec753bbe4f7c0b2e4018b9fa9e694237ac76b1f9fa7f89e44346d91123"} Oct 31 15:14:04.203602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.203560 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb" Oct 31 15:14:04.213859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.213841 4399 scope.go:122] "RemoveContainer" containerID="762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9" Oct 31 15:14:04.214135 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:04.214117 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9\": container with ID starting with 762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9 not found: ID does not exist" containerID="762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9" Oct 31 15:14:04.214202 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.214143 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9"} err="failed to get container status \"762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9\": rpc error: code = NotFound desc = could not find container \"762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9\": container with ID starting with 762562ed056fe25b68278e09b778caa8df50e06508b4fcbb703b0663e3bb6ca9 not found: ID does not exist" Oct 31 15:14:04.214202 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.214161 4399 scope.go:122] "RemoveContainer" containerID="284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703" Oct 31 15:14:04.224200 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.224186 4399 scope.go:122] "RemoveContainer" containerID="284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703" Oct 31 15:14:04.224451 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:04.224435 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703\": container with ID starting with 284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703 not found: ID does not exist" containerID="284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703" Oct 31 15:14:04.224496 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.224457 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703"} err="failed to get container status \"284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703\": rpc error: code = NotFound desc = could not find container \"284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703\": container with ID starting with 284d8e9a5d4358976f52274a39157f1fa9795e1f95c13ecb81bc46fa5e243703 not found: ID does not exist" Oct 31 15:14:04.365942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.365910 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm"] Oct 31 15:14:04.414538 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.414506 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-j7xnm"] Oct 31 15:14:04.525830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.525691 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb"] Oct 31 15:14:04.529067 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.529039 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5f17a1-openshift-default-597bcb485c-sgjsb"] Oct 31 15:14:04.897114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.897019 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" path="/var/lib/kubelet/pods/954d2d01-a14b-40b4-8f26-469bf2909f91/volumes" Oct 31 15:14:04.897635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:04.897611 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" path="/var/lib/kubelet/pods/b9461a39-85f3-43c8-a510-d8d8a184e9fa/volumes" Oct 31 15:14:16.773796 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:16.773760 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn"] Oct 31 15:14:16.774282 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:16.773983 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" podUID="98694ba7-d718-4754-aec2-be70a73ddfaf" containerName="maas-api" containerID="cri-o://645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3" gracePeriod=30 Oct 31 15:14:17.131784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.131687 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:14:17.247732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.237245 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98694ba7-d718-4754-aec2-be70a73ddfaf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98694ba7-d718-4754-aec2-be70a73ddfaf-openshift-service-ca\") pod \"98694ba7-d718-4754-aec2-be70a73ddfaf\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " Oct 31 15:14:17.247732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.237312 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-metrics-certs\") pod \"98694ba7-d718-4754-aec2-be70a73ddfaf\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " Oct 31 15:14:17.247732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.237483 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98694ba7-d718-4754-aec2-be70a73ddfaf-kube-api-access-l7ftn\" (UniqueName: \"kubernetes.io/projected/98694ba7-d718-4754-aec2-be70a73ddfaf-kube-api-access-l7ftn\") pod \"98694ba7-d718-4754-aec2-be70a73ddfaf\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " Oct 31 15:14:17.247732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.237544 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls\") pod \"98694ba7-d718-4754-aec2-be70a73ddfaf\" (UID: \"98694ba7-d718-4754-aec2-be70a73ddfaf\") " Oct 31 15:14:17.247732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.242069 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98694ba7-d718-4754-aec2-be70a73ddfaf-openshift-service-ca" pod "98694ba7-d718-4754-aec2-be70a73ddfaf" (UID: "98694ba7-d718-4754-aec2-be70a73ddfaf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:17.274720 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.266433 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98694ba7-d718-4754-aec2-be70a73ddfaf-kube-api-access-l7ftn" pod "98694ba7-d718-4754-aec2-be70a73ddfaf" (UID: "98694ba7-d718-4754-aec2-be70a73ddfaf"). InnerVolumeSpecName "kube-api-access-l7ftn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:17.308506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.308467 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls" pod "98694ba7-d718-4754-aec2-be70a73ddfaf" (UID: "98694ba7-d718-4754-aec2-be70a73ddfaf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:14:17.317559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.317505 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-metrics-certs" pod "98694ba7-d718-4754-aec2-be70a73ddfaf" (UID: "98694ba7-d718-4754-aec2-be70a73ddfaf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:14:17.328200 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.328173 4399 generic.go:356] "Generic (PLEG): container finished" podID="98694ba7-d718-4754-aec2-be70a73ddfaf" containerID="645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3" exitCode=0 Oct 31 15:14:17.328452 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.328399 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" event={"ID":"98694ba7-d718-4754-aec2-be70a73ddfaf","Type":"ContainerDied","Data":"645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3"} Oct 31 15:14:17.328537 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.328528 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" event={"ID":"98694ba7-d718-4754-aec2-be70a73ddfaf","Type":"ContainerDied","Data":"c1bd5e6435a677a5512ac988e44ebeb91f881e90d5adc179101da6ad0e83fa55"} Oct 31 15:14:17.328592 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.328561 4399 scope.go:122] "RemoveContainer" containerID="645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3" Oct 31 15:14:17.328745 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.328688 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn" Oct 31 15:14:17.340906 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.340876 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98694ba7-d718-4754-aec2-be70a73ddfaf-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:17.341050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.340912 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:17.341050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.340932 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7ftn\" (UniqueName: \"kubernetes.io/projected/98694ba7-d718-4754-aec2-be70a73ddfaf-kube-api-access-l7ftn\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:17.341050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.340950 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98694ba7-d718-4754-aec2-be70a73ddfaf-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:17.344152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.344127 4399 scope.go:122] "RemoveContainer" containerID="645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3" Oct 31 15:14:17.344538 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:17.344511 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3\": container with ID starting with 645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3 not found: ID does not exist" containerID="645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3" Oct 31 15:14:17.344636 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.344551 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3"} err="failed to get container status \"645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3\": rpc error: code = NotFound desc = could not find container \"645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3\": container with ID starting with 645022fd872ffb84a0aec1f708ad62ca9b866e8801830ab55fb81181429942e3 not found: ID does not exist" Oct 31 15:14:17.510688 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.510618 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn"] Oct 31 15:14:17.514803 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:17.514777 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-5695787b-77d88cdb7d-768nn"] Oct 31 15:14:18.922113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:18.922080 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98694ba7-d718-4754-aec2-be70a73ddfaf" path="/var/lib/kubelet/pods/98694ba7-d718-4754-aec2-be70a73ddfaf/volumes" Oct 31 15:14:24.573999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:24.573963 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg"] Oct 31 15:14:24.574456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:24.574248 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" containerID="cri-o://8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5" gracePeriod=30 Oct 31 15:14:24.585125 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:24.585098 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz"] Oct 31 15:14:24.585294 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:24.585275 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" containerID="cri-o://18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700" gracePeriod=30 Oct 31 15:14:24.602938 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:24.602910 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7"] Oct 31 15:14:24.603141 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:24.603118 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" containerID="cri-o://1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121" gracePeriod=30 Oct 31 15:14:24.622794 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:24.622767 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2"] Oct 31 15:14:24.624070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:24.624042 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" containerID="cri-o://90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5" gracePeriod=30 Oct 31 15:14:26.054504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:26.054092 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:14:26.054504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:26.054165 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:26.212070 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:26.212023 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:14:26.738288 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:26.738253 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6"] Oct 31 15:14:26.738548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:26.738505 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" containerID="cri-o://8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367" gracePeriod=30 Oct 31 15:14:27.427587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:27.427550 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:14:27.428010 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:27.427612 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:28.085896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:28.085864 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:14:28.086075 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:28.085926 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:28.210223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:28.210193 4399 patch_prober.go:29] interesting pod/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 31 15:14:28.210391 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:28.210240 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:14:29.907126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.907031 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:14:29.991545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991521 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-credential-socket\") pod \"ba7a7f41-b0db-48f9-bc23-153d28d62304\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " Oct 31 15:14:29.991636 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991555 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-certs\") pod \"ba7a7f41-b0db-48f9-bc23-153d28d62304\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " Oct 31 15:14:29.991636 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991587 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-data\") pod \"ba7a7f41-b0db-48f9-bc23-153d28d62304\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " Oct 31 15:14:29.991761 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991682 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-token\") pod \"ba7a7f41-b0db-48f9-bc23-153d28d62304\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " Oct 31 15:14:29.991824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991786 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-envoy\") pod \"ba7a7f41-b0db-48f9-bc23-153d28d62304\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " Oct 31 15:14:29.991879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991818 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-certs" pod "ba7a7f41-b0db-48f9-bc23-153d28d62304" (UID: "ba7a7f41-b0db-48f9-bc23-153d28d62304"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:29.991879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991850 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-kube-api-access-9s9fc\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-kube-api-access-9s9fc\") pod \"ba7a7f41-b0db-48f9-bc23-153d28d62304\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " Oct 31 15:14:29.991962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991916 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-podinfo\") pod \"ba7a7f41-b0db-48f9-bc23-153d28d62304\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " Oct 31 15:14:29.991962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991948 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-credential-socket" pod "ba7a7f41-b0db-48f9-bc23-153d28d62304" (UID: "ba7a7f41-b0db-48f9-bc23-153d28d62304"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:29.992076 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991975 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-data" pod "ba7a7f41-b0db-48f9-bc23-153d28d62304" (UID: "ba7a7f41-b0db-48f9-bc23-153d28d62304"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:29.992076 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.991997 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-socket\") pod \"ba7a7f41-b0db-48f9-bc23-153d28d62304\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " Oct 31 15:14:29.992076 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.992040 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba7a7f41-b0db-48f9-bc23-153d28d62304-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7f41-b0db-48f9-bc23-153d28d62304-istiod-ca-cert\") pod \"ba7a7f41-b0db-48f9-bc23-153d28d62304\" (UID: \"ba7a7f41-b0db-48f9-bc23-153d28d62304\") " Oct 31 15:14:29.992253 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.992232 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-socket" pod "ba7a7f41-b0db-48f9-bc23-153d28d62304" (UID: "ba7a7f41-b0db-48f9-bc23-153d28d62304"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:29.992571 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.992524 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba7a7f41-b0db-48f9-bc23-153d28d62304-istiod-ca-cert" pod "ba7a7f41-b0db-48f9-bc23-153d28d62304" (UID: "ba7a7f41-b0db-48f9-bc23-153d28d62304"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:29.993016 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.992962 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:29.993016 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.992992 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba7a7f41-b0db-48f9-bc23-153d28d62304-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:29.993016 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.993007 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:29.993169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.993021 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:29.993169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.993035 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:29.994247 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.994223 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-podinfo" pod "ba7a7f41-b0db-48f9-bc23-153d28d62304" (UID: "ba7a7f41-b0db-48f9-bc23-153d28d62304"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:29.994462 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.994440 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-token" pod "ba7a7f41-b0db-48f9-bc23-153d28d62304" (UID: "ba7a7f41-b0db-48f9-bc23-153d28d62304"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:29.994671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.994657 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-kube-api-access-9s9fc" pod "ba7a7f41-b0db-48f9-bc23-153d28d62304" (UID: "ba7a7f41-b0db-48f9-bc23-153d28d62304"). InnerVolumeSpecName "kube-api-access-9s9fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:29.994778 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:29.994762 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-envoy" pod "ba7a7f41-b0db-48f9-bc23-153d28d62304" (UID: "ba7a7f41-b0db-48f9-bc23-153d28d62304"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.001912 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.001894 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:14:30.004416 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.004399 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:14:30.008239 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.008220 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:14:30.094101 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094067 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-certs\") pod \"a25234fe-9717-4f3c-874f-cac85c87cf6c\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " Oct 31 15:14:30.094279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094115 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-envoy\") pod \"a25234fe-9717-4f3c-874f-cac85c87cf6c\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " Oct 31 15:14:30.094279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094165 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-envoy\") pod \"48578e0d-a009-4589-bf9a-54f4acdd7416\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " Oct 31 15:14:30.094279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094220 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-istio-token\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-istio-token\") pod \"2c9e388b-1fab-43da-8991-1fe0df782480\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " Oct 31 15:14:30.094279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094260 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-kube-api-access-brgnx\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-kube-api-access-brgnx\") pod \"a25234fe-9717-4f3c-874f-cac85c87cf6c\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " Oct 31 15:14:30.094502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094298 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-socket\") pod \"48578e0d-a009-4589-bf9a-54f4acdd7416\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " Oct 31 15:14:30.094502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094343 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-certs\") pod \"2c9e388b-1fab-43da-8991-1fe0df782480\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " Oct 31 15:14:30.094502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094375 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-envoy\") pod \"2c9e388b-1fab-43da-8991-1fe0df782480\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " Oct 31 15:14:30.094502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094401 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-socket\") pod \"2c9e388b-1fab-43da-8991-1fe0df782480\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " Oct 31 15:14:30.094502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094427 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-token\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-token\") pod \"a25234fe-9717-4f3c-874f-cac85c87cf6c\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " Oct 31 15:14:30.094502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094470 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-data\") pod \"2c9e388b-1fab-43da-8991-1fe0df782480\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " Oct 31 15:14:30.094502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094496 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9e388b-1fab-43da-8991-1fe0df782480-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9e388b-1fab-43da-8991-1fe0df782480-istiod-ca-cert\") pod \"2c9e388b-1fab-43da-8991-1fe0df782480\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " Oct 31 15:14:30.094811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094576 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-kube-api-access-tvsc7\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-kube-api-access-tvsc7\") pod \"2c9e388b-1fab-43da-8991-1fe0df782480\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " Oct 31 15:14:30.094811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094618 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-socket\") pod \"a25234fe-9717-4f3c-874f-cac85c87cf6c\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " Oct 31 15:14:30.094811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094681 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-data\") pod \"a25234fe-9717-4f3c-874f-cac85c87cf6c\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " Oct 31 15:14:30.094811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094758 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48578e0d-a009-4589-bf9a-54f4acdd7416-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48578e0d-a009-4589-bf9a-54f4acdd7416-istiod-ca-cert\") pod \"48578e0d-a009-4589-bf9a-54f4acdd7416\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " Oct 31 15:14:30.094811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094789 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-podinfo\") pod \"a25234fe-9717-4f3c-874f-cac85c87cf6c\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " Oct 31 15:14:30.094999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094829 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-kube-api-access-lxfx2\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-kube-api-access-lxfx2\") pod \"48578e0d-a009-4589-bf9a-54f4acdd7416\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " Oct 31 15:14:30.094999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094886 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-certs\") pod \"48578e0d-a009-4589-bf9a-54f4acdd7416\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " Oct 31 15:14:30.094999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094930 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-credential-socket\") pod \"2c9e388b-1fab-43da-8991-1fe0df782480\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " Oct 31 15:14:30.094999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094954 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-podinfo\") pod \"48578e0d-a009-4589-bf9a-54f4acdd7416\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " Oct 31 15:14:30.094999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.094976 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c9e388b-1fab-43da-8991-1fe0df782480-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9e388b-1fab-43da-8991-1fe0df782480-istio-podinfo\") pod \"2c9e388b-1fab-43da-8991-1fe0df782480\" (UID: \"2c9e388b-1fab-43da-8991-1fe0df782480\") " Oct 31 15:14:30.095166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.095012 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a25234fe-9717-4f3c-874f-cac85c87cf6c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25234fe-9717-4f3c-874f-cac85c87cf6c-istiod-ca-cert\") pod \"a25234fe-9717-4f3c-874f-cac85c87cf6c\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " Oct 31 15:14:30.095166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.095030 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-credential-socket\") pod \"48578e0d-a009-4589-bf9a-54f4acdd7416\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " Oct 31 15:14:30.095166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.095053 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-token\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-token\") pod \"48578e0d-a009-4589-bf9a-54f4acdd7416\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " Oct 31 15:14:30.095166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.095077 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-credential-socket\") pod \"a25234fe-9717-4f3c-874f-cac85c87cf6c\" (UID: \"a25234fe-9717-4f3c-874f-cac85c87cf6c\") " Oct 31 15:14:30.095166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.095094 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-data\") pod \"48578e0d-a009-4589-bf9a-54f4acdd7416\" (UID: \"48578e0d-a009-4589-bf9a-54f4acdd7416\") " Oct 31 15:14:30.095758 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.095734 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.095862 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.095763 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.095862 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.095780 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s9fc\" (UniqueName: \"kubernetes.io/projected/ba7a7f41-b0db-48f9-bc23-153d28d62304-kube-api-access-9s9fc\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.095862 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.095790 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba7a7f41-b0db-48f9-bc23-153d28d62304-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.096331 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-socket" pod "48578e0d-a009-4589-bf9a-54f4acdd7416" (UID: "48578e0d-a009-4589-bf9a-54f4acdd7416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.096433 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-data" pod "48578e0d-a009-4589-bf9a-54f4acdd7416" (UID: "48578e0d-a009-4589-bf9a-54f4acdd7416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.096483 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-credential-socket" pod "2c9e388b-1fab-43da-8991-1fe0df782480" (UID: "2c9e388b-1fab-43da-8991-1fe0df782480"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.096578 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-certs" pod "a25234fe-9717-4f3c-874f-cac85c87cf6c" (UID: "a25234fe-9717-4f3c-874f-cac85c87cf6c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.096608 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-credential-socket" pod "a25234fe-9717-4f3c-874f-cac85c87cf6c" (UID: "a25234fe-9717-4f3c-874f-cac85c87cf6c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.096763 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-socket" pod "a25234fe-9717-4f3c-874f-cac85c87cf6c" (UID: "a25234fe-9717-4f3c-874f-cac85c87cf6c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.096776 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a25234fe-9717-4f3c-874f-cac85c87cf6c-istiod-ca-cert" pod "a25234fe-9717-4f3c-874f-cac85c87cf6c" (UID: "a25234fe-9717-4f3c-874f-cac85c87cf6c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:30.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.096973 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-data" pod "a25234fe-9717-4f3c-874f-cac85c87cf6c" (UID: "a25234fe-9717-4f3c-874f-cac85c87cf6c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.097266 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48578e0d-a009-4589-bf9a-54f4acdd7416-istiod-ca-cert" pod "48578e0d-a009-4589-bf9a-54f4acdd7416" (UID: "48578e0d-a009-4589-bf9a-54f4acdd7416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:30.102137 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.101966 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-podinfo" pod "a25234fe-9717-4f3c-874f-cac85c87cf6c" (UID: "a25234fe-9717-4f3c-874f-cac85c87cf6c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:30.102332 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.102279 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-certs" pod "48578e0d-a009-4589-bf9a-54f4acdd7416" (UID: "48578e0d-a009-4589-bf9a-54f4acdd7416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.102651 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.102621 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-socket" pod "2c9e388b-1fab-43da-8991-1fe0df782480" (UID: "2c9e388b-1fab-43da-8991-1fe0df782480"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.102800 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.102716 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-kube-api-access-brgnx" pod "a25234fe-9717-4f3c-874f-cac85c87cf6c" (UID: "a25234fe-9717-4f3c-874f-cac85c87cf6c"). InnerVolumeSpecName "kube-api-access-brgnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.102881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.102788 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-envoy" pod "a25234fe-9717-4f3c-874f-cac85c87cf6c" (UID: "a25234fe-9717-4f3c-874f-cac85c87cf6c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.102881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.102869 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-certs" pod "2c9e388b-1fab-43da-8991-1fe0df782480" (UID: "2c9e388b-1fab-43da-8991-1fe0df782480"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.102989 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.102902 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-istio-token" pod "2c9e388b-1fab-43da-8991-1fe0df782480" (UID: "2c9e388b-1fab-43da-8991-1fe0df782480"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.103135 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.103101 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-credential-socket" pod "48578e0d-a009-4589-bf9a-54f4acdd7416" (UID: "48578e0d-a009-4589-bf9a-54f4acdd7416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.103299 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.103279 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-data" pod "2c9e388b-1fab-43da-8991-1fe0df782480" (UID: "2c9e388b-1fab-43da-8991-1fe0df782480"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.104937 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.103458 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-envoy" pod "48578e0d-a009-4589-bf9a-54f4acdd7416" (UID: "48578e0d-a009-4589-bf9a-54f4acdd7416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.104937 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.103680 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-podinfo" pod "48578e0d-a009-4589-bf9a-54f4acdd7416" (UID: "48578e0d-a009-4589-bf9a-54f4acdd7416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:30.104937 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.103861 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9e388b-1fab-43da-8991-1fe0df782480-istiod-ca-cert" pod "2c9e388b-1fab-43da-8991-1fe0df782480" (UID: "2c9e388b-1fab-43da-8991-1fe0df782480"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:30.105195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.105165 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-kube-api-access-lxfx2" pod "48578e0d-a009-4589-bf9a-54f4acdd7416" (UID: "48578e0d-a009-4589-bf9a-54f4acdd7416"). InnerVolumeSpecName "kube-api-access-lxfx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.105732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.105673 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c9e388b-1fab-43da-8991-1fe0df782480-istio-podinfo" pod "2c9e388b-1fab-43da-8991-1fe0df782480" (UID: "2c9e388b-1fab-43da-8991-1fe0df782480"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:30.106051 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.106006 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-envoy" pod "2c9e388b-1fab-43da-8991-1fe0df782480" (UID: "2c9e388b-1fab-43da-8991-1fe0df782480"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:30.107109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.107081 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-token" pod "48578e0d-a009-4589-bf9a-54f4acdd7416" (UID: "48578e0d-a009-4589-bf9a-54f4acdd7416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.107309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.107284 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-token" pod "a25234fe-9717-4f3c-874f-cac85c87cf6c" (UID: "a25234fe-9717-4f3c-874f-cac85c87cf6c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.108052 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.108025 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-kube-api-access-tvsc7" pod "2c9e388b-1fab-43da-8991-1fe0df782480" (UID: "2c9e388b-1fab-43da-8991-1fe0df782480"). InnerVolumeSpecName "kube-api-access-tvsc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:30.197717 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197671 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48578e0d-a009-4589-bf9a-54f4acdd7416-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197742 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197757 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxfx2\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-kube-api-access-lxfx2\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197769 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197778 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197787 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197795 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c9e388b-1fab-43da-8991-1fe0df782480-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197803 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a25234fe-9717-4f3c-874f-cac85c87cf6c-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197812 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197820 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197828 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197836 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197845 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197852 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197861 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197871 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197879 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brgnx\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-kube-api-access-brgnx\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197887 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48578e0d-a009-4589-bf9a-54f4acdd7416-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197895 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.197895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197903 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.198968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197911 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.198968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197920 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.198968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197928 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c9e388b-1fab-43da-8991-1fe0df782480-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.198968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197937 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c9e388b-1fab-43da-8991-1fe0df782480-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.198968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197946 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvsc7\" (UniqueName: \"kubernetes.io/projected/2c9e388b-1fab-43da-8991-1fe0df782480-kube-api-access-tvsc7\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.198968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197955 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.198968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.197963 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a25234fe-9717-4f3c-874f-cac85c87cf6c-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:30.480974 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.480888 4399 generic.go:356] "Generic (PLEG): container finished" podID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerID="90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5" exitCode=0 Oct 31 15:14:30.481165 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.480998 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" event={"ID":"ba7a7f41-b0db-48f9-bc23-153d28d62304","Type":"ContainerDied","Data":"90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5"} Oct 31 15:14:30.481165 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.480999 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" Oct 31 15:14:30.481165 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.481096 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2" event={"ID":"ba7a7f41-b0db-48f9-bc23-153d28d62304","Type":"ContainerDied","Data":"52b0077fdd5a3008ae3439e1a52e1652f8ff7b168bf3cc372246a1315497e542"} Oct 31 15:14:30.481165 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.481130 4399 scope.go:122] "RemoveContainer" containerID="90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5" Oct 31 15:14:30.482587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.482561 4399 generic.go:356] "Generic (PLEG): container finished" podID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerID="18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700" exitCode=0 Oct 31 15:14:30.482915 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.482637 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" event={"ID":"48578e0d-a009-4589-bf9a-54f4acdd7416","Type":"ContainerDied","Data":"18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700"} Oct 31 15:14:30.482915 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.482677 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" event={"ID":"48578e0d-a009-4589-bf9a-54f4acdd7416","Type":"ContainerDied","Data":"8aef888d300a11274377e8ed69245f1815cff998ff022babe71ddd276d4d78e2"} Oct 31 15:14:30.482915 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.482788 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz" Oct 31 15:14:30.492797 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.492776 4399 generic.go:356] "Generic (PLEG): container finished" podID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerID="1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121" exitCode=0 Oct 31 15:14:30.493035 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.493018 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" Oct 31 15:14:30.493336 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.493313 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" event={"ID":"a25234fe-9717-4f3c-874f-cac85c87cf6c","Type":"ContainerDied","Data":"1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121"} Oct 31 15:14:30.493743 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.493715 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7" event={"ID":"a25234fe-9717-4f3c-874f-cac85c87cf6c","Type":"ContainerDied","Data":"5ce65a9bc5e42c80332a2cc753835d5cdf2808310b0d52406c2aa2cd8ba690d3"} Oct 31 15:14:30.494178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.494160 4399 scope.go:122] "RemoveContainer" containerID="90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5" Oct 31 15:14:30.494842 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:30.494813 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5\": container with ID starting with 90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5 not found: ID does not exist" containerID="90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5" Oct 31 15:14:30.495947 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.495919 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5"} err="failed to get container status \"90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5\": rpc error: code = NotFound desc = could not find container \"90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5\": container with ID starting with 90e8af0dcac769a20e750d3dc07d1c03778ef45400dd6d3ad43dec68986541b5 not found: ID does not exist" Oct 31 15:14:30.496259 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.496238 4399 scope.go:122] "RemoveContainer" containerID="18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700" Oct 31 15:14:30.504783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.500350 4399 generic.go:356] "Generic (PLEG): container finished" podID="2c9e388b-1fab-43da-8991-1fe0df782480" containerID="8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5" exitCode=0 Oct 31 15:14:30.504783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.500448 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" event={"ID":"2c9e388b-1fab-43da-8991-1fe0df782480","Type":"ContainerDied","Data":"8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5"} Oct 31 15:14:30.504783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.500483 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" event={"ID":"2c9e388b-1fab-43da-8991-1fe0df782480","Type":"ContainerDied","Data":"3d0305d90a9e9a5655e37eb87f0c031654885a881c66271522bdb7c925648f68"} Oct 31 15:14:30.504783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.500550 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg" Oct 31 15:14:30.513310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.513296 4399 scope.go:122] "RemoveContainer" containerID="18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700" Oct 31 15:14:30.513615 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:30.513599 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700\": container with ID starting with 18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700 not found: ID does not exist" containerID="18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700" Oct 31 15:14:30.513687 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.513624 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700"} err="failed to get container status \"18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700\": rpc error: code = NotFound desc = could not find container \"18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700\": container with ID starting with 18c6b07dd920beec07fb535c45635c18d2e0fb8daecd1c302657352608c60700 not found: ID does not exist" Oct 31 15:14:30.513687 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.513647 4399 scope.go:122] "RemoveContainer" containerID="1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121" Oct 31 15:14:30.523475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.523460 4399 scope.go:122] "RemoveContainer" containerID="1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121" Oct 31 15:14:30.523787 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:30.523766 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121\": container with ID starting with 1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121 not found: ID does not exist" containerID="1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121" Oct 31 15:14:30.523858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.523791 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121"} err="failed to get container status \"1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121\": rpc error: code = NotFound desc = could not find container \"1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121\": container with ID starting with 1ae4bb43fe969875a92cfa986f33ff1ad7ad88e55c23562909ee49b37395a121 not found: ID does not exist" Oct 31 15:14:30.523858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.523806 4399 scope.go:122] "RemoveContainer" containerID="8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5" Oct 31 15:14:30.532804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.532788 4399 scope.go:122] "RemoveContainer" containerID="8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5" Oct 31 15:14:30.533051 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:30.533032 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5\": container with ID starting with 8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5 not found: ID does not exist" containerID="8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5" Oct 31 15:14:30.533115 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.533056 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5"} err="failed to get container status \"8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5\": rpc error: code = NotFound desc = could not find container \"8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5\": container with ID starting with 8dbab296a5c380399b05484ee4bee821df92971d3f93d3c58976ac4eb7e6b5c5 not found: ID does not exist" Oct 31 15:14:30.706389 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.706359 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2"] Oct 31 15:14:30.709480 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.709461 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-bf7l2"] Oct 31 15:14:30.818750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.818678 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg"] Oct 31 15:14:30.825461 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.825437 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-5p9bg"] Oct 31 15:14:30.898189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.898163 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" path="/var/lib/kubelet/pods/2c9e388b-1fab-43da-8991-1fe0df782480/volumes" Oct 31 15:14:30.898658 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.898643 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" path="/var/lib/kubelet/pods/ba7a7f41-b0db-48f9-bc23-153d28d62304/volumes" Oct 31 15:14:30.933469 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.933445 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz"] Oct 31 15:14:30.936566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:30.936546 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gg7qz"] Oct 31 15:14:31.046910 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:31.046884 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7"] Oct 31 15:14:31.049774 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:31.049755 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5695787b-openshift-default-578b488fb5-gmwm7"] Oct 31 15:14:32.016855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.016796 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:14:32.129030 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129001 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-data\") pod \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " Oct 31 15:14:32.129180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129072 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-envoy\") pod \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " Oct 31 15:14:32.129180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129139 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-token\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-token\") pod \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " Oct 31 15:14:32.129300 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129188 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-credential-socket\") pod \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " Oct 31 15:14:32.129300 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129203 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-data" pod "7cae3859-4f99-4a41-88f9-fc6ae17e301c" (UID: "7cae3859-4f99-4a41-88f9-fc6ae17e301c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:32.129300 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129218 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-socket\") pod \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " Oct 31 15:14:32.129454 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129298 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-kube-api-access-2ngjs\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-kube-api-access-2ngjs\") pod \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " Oct 31 15:14:32.129454 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129415 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istiod-ca-cert\") pod \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " Oct 31 15:14:32.129454 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129447 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-certs\") pod \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " Oct 31 15:14:32.129619 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129476 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-podinfo\") pod \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\" (UID: \"7cae3859-4f99-4a41-88f9-fc6ae17e301c\") " Oct 31 15:14:32.129619 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129495 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-credential-socket" pod "7cae3859-4f99-4a41-88f9-fc6ae17e301c" (UID: "7cae3859-4f99-4a41-88f9-fc6ae17e301c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:32.129619 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129514 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-socket" pod "7cae3859-4f99-4a41-88f9-fc6ae17e301c" (UID: "7cae3859-4f99-4a41-88f9-fc6ae17e301c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:32.129803 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129753 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-certs" pod "7cae3859-4f99-4a41-88f9-fc6ae17e301c" (UID: "7cae3859-4f99-4a41-88f9-fc6ae17e301c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:32.129889 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.129858 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istiod-ca-cert" pod "7cae3859-4f99-4a41-88f9-fc6ae17e301c" (UID: "7cae3859-4f99-4a41-88f9-fc6ae17e301c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:32.130147 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.130123 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:32.130440 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.130149 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:32.130440 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.130163 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:32.130440 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.130173 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:32.130440 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.130187 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:32.131440 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.131419 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-token" pod "7cae3859-4f99-4a41-88f9-fc6ae17e301c" (UID: "7cae3859-4f99-4a41-88f9-fc6ae17e301c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:32.131843 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.131823 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-envoy" pod "7cae3859-4f99-4a41-88f9-fc6ae17e301c" (UID: "7cae3859-4f99-4a41-88f9-fc6ae17e301c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:14:32.131923 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.131883 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-kube-api-access-2ngjs" pod "7cae3859-4f99-4a41-88f9-fc6ae17e301c" (UID: "7cae3859-4f99-4a41-88f9-fc6ae17e301c"). InnerVolumeSpecName "kube-api-access-2ngjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:32.131961 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.131920 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-podinfo" pod "7cae3859-4f99-4a41-88f9-fc6ae17e301c" (UID: "7cae3859-4f99-4a41-88f9-fc6ae17e301c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:14:32.231364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.231331 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ngjs\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-kube-api-access-2ngjs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:32.231364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.231368 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:32.231625 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.231387 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:32.231625 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.231402 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cae3859-4f99-4a41-88f9-fc6ae17e301c-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:32.519330 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.519294 4399 generic.go:356] "Generic (PLEG): container finished" podID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerID="8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367" exitCode=0 Oct 31 15:14:32.519510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.519343 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" Oct 31 15:14:32.519510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.519344 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" event={"ID":"7cae3859-4f99-4a41-88f9-fc6ae17e301c","Type":"ContainerDied","Data":"8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367"} Oct 31 15:14:32.519510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.519472 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" event={"ID":"7cae3859-4f99-4a41-88f9-fc6ae17e301c","Type":"ContainerDied","Data":"47f2ca91f67ffee51d9d282d705eb9f8b9d104f75b1349dcdcb4c1f3d1d2a765"} Oct 31 15:14:32.519510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.519503 4399 scope.go:122] "RemoveContainer" containerID="8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367" Oct 31 15:14:32.530374 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.530356 4399 scope.go:122] "RemoveContainer" containerID="8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367" Oct 31 15:14:32.530629 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:32.530612 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367\": container with ID starting with 8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367 not found: ID does not exist" containerID="8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367" Oct 31 15:14:32.530666 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.530636 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367"} err="failed to get container status \"8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367\": rpc error: code = NotFound desc = could not find container \"8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367\": container with ID starting with 8da7d95271b27e37c1f9e0c3001be298b3742b4fe07ad1258ea35b4213844367 not found: ID does not exist" Oct 31 15:14:32.684689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.684658 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6"] Oct 31 15:14:32.733442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.733408 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6"] Oct 31 15:14:32.897832 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.897752 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" path="/var/lib/kubelet/pods/48578e0d-a009-4589-bf9a-54f4acdd7416/volumes" Oct 31 15:14:32.898209 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.898195 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" path="/var/lib/kubelet/pods/7cae3859-4f99-4a41-88f9-fc6ae17e301c/volumes" Oct 31 15:14:32.898601 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:32.898587 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" path="/var/lib/kubelet/pods/a25234fe-9717-4f3c-874f-cac85c87cf6c/volumes" Oct 31 15:14:33.001094 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:33.001066 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:14:33.001255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:33.001113 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-794r6" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:14:37.151737 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:37.151692 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs"] Oct 31 15:14:37.258464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:37.258425 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z"] Oct 31 15:14:42.152193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.152122 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" podUID="78e95b96-7906-4dbe-8bd2-72d115517ae8" containerName="payload-processing" containerID="cri-o://940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111" gracePeriod=25 Oct 31 15:14:42.259161 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.259120 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" podUID="2d0860d0-4608-4a18-a0a1-fed76403af03" containerName="payload-pre-processing" containerID="cri-o://b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3" gracePeriod=25 Oct 31 15:14:42.424434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.424405 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:14:42.494788 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.494747 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e95b96-7906-4dbe-8bd2-72d115517ae8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e95b96-7906-4dbe-8bd2-72d115517ae8-plugins-config-volume\") pod \"78e95b96-7906-4dbe-8bd2-72d115517ae8\" (UID: \"78e95b96-7906-4dbe-8bd2-72d115517ae8\") " Oct 31 15:14:42.494788 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.494798 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e95b96-7906-4dbe-8bd2-72d115517ae8-kube-api-access-j8czs\" (UniqueName: \"kubernetes.io/projected/78e95b96-7906-4dbe-8bd2-72d115517ae8-kube-api-access-j8czs\") pod \"78e95b96-7906-4dbe-8bd2-72d115517ae8\" (UID: \"78e95b96-7906-4dbe-8bd2-72d115517ae8\") " Oct 31 15:14:42.524587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.524554 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e95b96-7906-4dbe-8bd2-72d115517ae8-plugins-config-volume" pod "78e95b96-7906-4dbe-8bd2-72d115517ae8" (UID: "78e95b96-7906-4dbe-8bd2-72d115517ae8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:42.526587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.526544 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e95b96-7906-4dbe-8bd2-72d115517ae8-kube-api-access-j8czs" pod "78e95b96-7906-4dbe-8bd2-72d115517ae8" (UID: "78e95b96-7906-4dbe-8bd2-72d115517ae8"). InnerVolumeSpecName "kube-api-access-j8czs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:42.555456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.555433 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:14:42.589225 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.589197 4399 generic.go:356] "Generic (PLEG): container finished" podID="2d0860d0-4608-4a18-a0a1-fed76403af03" containerID="b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3" exitCode=0 Oct 31 15:14:42.589363 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.589254 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" Oct 31 15:14:42.589363 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.589281 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" event={"ID":"2d0860d0-4608-4a18-a0a1-fed76403af03","Type":"ContainerDied","Data":"b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3"} Oct 31 15:14:42.589363 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.589352 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z" event={"ID":"2d0860d0-4608-4a18-a0a1-fed76403af03","Type":"ContainerDied","Data":"15fc6577534ac4160735b6b7e0da7ffcccaa600114cf4d7f43a9adb4c44e96ab"} Oct 31 15:14:42.589521 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.589387 4399 scope.go:122] "RemoveContainer" containerID="b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3" Oct 31 15:14:42.591007 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.590981 4399 generic.go:356] "Generic (PLEG): container finished" podID="78e95b96-7906-4dbe-8bd2-72d115517ae8" containerID="940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111" exitCode=0 Oct 31 15:14:42.591109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.591030 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" event={"ID":"78e95b96-7906-4dbe-8bd2-72d115517ae8","Type":"ContainerDied","Data":"940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111"} Oct 31 15:14:42.591109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.591067 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" event={"ID":"78e95b96-7906-4dbe-8bd2-72d115517ae8","Type":"ContainerDied","Data":"a00280f1ab1d28b5e6d8a2759931d6a4baa090a803b8afc47d949ce9f6d0c7fb"} Oct 31 15:14:42.591215 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.591132 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs" Oct 31 15:14:42.597011 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.596988 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78e95b96-7906-4dbe-8bd2-72d115517ae8-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:42.597011 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.597010 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8czs\" (UniqueName: \"kubernetes.io/projected/78e95b96-7906-4dbe-8bd2-72d115517ae8-kube-api-access-j8czs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:42.600590 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.600571 4399 scope.go:122] "RemoveContainer" containerID="b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3" Oct 31 15:14:42.600868 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:42.600851 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3\": container with ID starting with b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3 not found: ID does not exist" containerID="b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3" Oct 31 15:14:42.600946 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.600874 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3"} err="failed to get container status \"b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3\": rpc error: code = NotFound desc = could not find container \"b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3\": container with ID starting with b1a9f66e530132a2a7f631d0503e331237bc284a33a8f04b6f6d66e721a835c3 not found: ID does not exist" Oct 31 15:14:42.600946 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.600911 4399 scope.go:122] "RemoveContainer" containerID="940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111" Oct 31 15:14:42.609903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.609887 4399 scope.go:122] "RemoveContainer" containerID="940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111" Oct 31 15:14:42.610183 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:42.610163 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111\": container with ID starting with 940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111 not found: ID does not exist" containerID="940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111" Oct 31 15:14:42.610251 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.610188 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111"} err="failed to get container status \"940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111\": rpc error: code = NotFound desc = could not find container \"940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111\": container with ID starting with 940e0f91208efcb0e9ad5e795771a7d9307a92b7eb1a319a591f7fea7fc2e111 not found: ID does not exist" Oct 31 15:14:42.698069 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.698032 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d0860d0-4608-4a18-a0a1-fed76403af03-kube-api-access-q62c2\" (UniqueName: \"kubernetes.io/projected/2d0860d0-4608-4a18-a0a1-fed76403af03-kube-api-access-q62c2\") pod \"2d0860d0-4608-4a18-a0a1-fed76403af03\" (UID: \"2d0860d0-4608-4a18-a0a1-fed76403af03\") " Oct 31 15:14:42.698335 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.698315 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d0860d0-4608-4a18-a0a1-fed76403af03-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0860d0-4608-4a18-a0a1-fed76403af03-plugins-config-volume\") pod \"2d0860d0-4608-4a18-a0a1-fed76403af03\" (UID: \"2d0860d0-4608-4a18-a0a1-fed76403af03\") " Oct 31 15:14:42.698687 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.698650 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0860d0-4608-4a18-a0a1-fed76403af03-plugins-config-volume" pod "2d0860d0-4608-4a18-a0a1-fed76403af03" (UID: "2d0860d0-4608-4a18-a0a1-fed76403af03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:14:42.699051 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.699025 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d0860d0-4608-4a18-a0a1-fed76403af03-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:42.700989 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.700953 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d0860d0-4608-4a18-a0a1-fed76403af03-kube-api-access-q62c2" pod "2d0860d0-4608-4a18-a0a1-fed76403af03" (UID: "2d0860d0-4608-4a18-a0a1-fed76403af03"). InnerVolumeSpecName "kube-api-access-q62c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:14:42.800169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.800143 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62c2\" (UniqueName: \"kubernetes.io/projected/2d0860d0-4608-4a18-a0a1-fed76403af03-kube-api-access-q62c2\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:14:42.831589 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.831559 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs"] Oct 31 15:14:42.836225 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.836186 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-94a889-5f9bc7d885-fftzs"] Oct 31 15:14:42.900511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:42.900479 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e95b96-7906-4dbe-8bd2-72d115517ae8" path="/var/lib/kubelet/pods/78e95b96-7906-4dbe-8bd2-72d115517ae8/volumes" Oct 31 15:14:43.013258 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:43.013171 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z"] Oct 31 15:14:43.016161 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:43.016138 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-94a889-66444d7f4c-6xw7z"] Oct 31 15:14:44.898186 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:44.898148 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d0860d0-4608-4a18-a0a1-fed76403af03" path="/var/lib/kubelet/pods/2d0860d0-4608-4a18-a0a1-fed76403af03/volumes" Oct 31 15:14:56.224749 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:14:56.224688 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:14:57.269114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:14:57.269084 4399 scope.go:122] "RemoveContainer" containerID="e310ca93f9547fc49d5645c83680c9ac301aa3ec32204f109dbc0ef75ded8431" Oct 31 15:15:00.305796 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.305762 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl"] Oct 31 15:15:00.306362 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306346 4399 cpu_manager.go:414] "containerMap: removing container" podUID="78e95b96-7906-4dbe-8bd2-72d115517ae8" containerName="payload-processing" Oct 31 15:15:00.306362 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306360 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e95b96-7906-4dbe-8bd2-72d115517ae8" containerName="payload-processing" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306377 4399 cpu_manager.go:414] "containerMap: removing container" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306383 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306392 4399 cpu_manager.go:414] "containerMap: removing container" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306397 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306408 4399 cpu_manager.go:414] "containerMap: removing container" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306413 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306428 4399 cpu_manager.go:414] "containerMap: removing container" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306432 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306447 4399 cpu_manager.go:414] "containerMap: removing container" podUID="2d0860d0-4608-4a18-a0a1-fed76403af03" containerName="payload-pre-processing" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306452 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d0860d0-4608-4a18-a0a1-fed76403af03" containerName="payload-pre-processing" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306461 4399 cpu_manager.go:414] "containerMap: removing container" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306465 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306472 4399 cpu_manager.go:414] "containerMap: removing container" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306476 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306484 4399 cpu_manager.go:414] "containerMap: removing container" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306488 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306503 4399 cpu_manager.go:414] "containerMap: removing container" podUID="98694ba7-d718-4754-aec2-be70a73ddfaf" containerName="maas-api" Oct 31 15:15:00.306504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306508 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98694ba7-d718-4754-aec2-be70a73ddfaf" containerName="maas-api" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306590 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9e388b-1fab-43da-8991-1fe0df782480" containerName="istio-proxy" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306599 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a25234fe-9717-4f3c-874f-cac85c87cf6c" containerName="istio-proxy" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306609 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d0860d0-4608-4a18-a0a1-fed76403af03" containerName="payload-pre-processing" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306616 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9461a39-85f3-43c8-a510-d8d8a184e9fa" containerName="istio-proxy" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306622 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="48578e0d-a009-4589-bf9a-54f4acdd7416" containerName="istio-proxy" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306628 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba7a7f41-b0db-48f9-bc23-153d28d62304" containerName="istio-proxy" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306635 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="98694ba7-d718-4754-aec2-be70a73ddfaf" containerName="maas-api" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306639 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e95b96-7906-4dbe-8bd2-72d115517ae8" containerName="payload-processing" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306646 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="954d2d01-a14b-40b4-8f26-469bf2909f91" containerName="istio-proxy" Oct 31 15:15:00.307437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.306650 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cae3859-4f99-4a41-88f9-fc6ae17e301c" containerName="istio-proxy" Oct 31 15:15:00.312050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.312025 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.370287 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.370249 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl"] Oct 31 15:15:00.414583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.414553 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-032b05-serving-cert\"" Oct 31 15:15:00.417220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.417190 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scvdw\" (UniqueName: \"kubernetes.io/projected/ac2df8a3-04d0-407f-a4e6-269f49073e0f-kube-api-access-scvdw\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.417391 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.417310 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2df8a3-04d0-407f-a4e6-269f49073e0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.417391 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.417366 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.417467 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.417408 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-metrics-certs\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.519134 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.519102 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-metrics-certs\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.519310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.519228 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scvdw\" (UniqueName: \"kubernetes.io/projected/ac2df8a3-04d0-407f-a4e6-269f49073e0f-kube-api-access-scvdw\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.519310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.519285 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2df8a3-04d0-407f-a4e6-269f49073e0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.519438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.519316 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.519438 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:00.519406 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-serving-cert: secret "maas-api-e2e-ipp-a-032b05-serving-cert" not found Oct 31 15:15:00.519542 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:00.519450 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls podName:ac2df8a3-04d0-407f-a4e6-269f49073e0f nodeName:}" failed. No retries permitted until 2026-10-31 15:15:01.019436075 +0000 UTC m=+1744.230677468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls") pod "maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" (UID: "ac2df8a3-04d0-407f-a4e6-269f49073e0f") : secret "maas-api-e2e-ipp-a-032b05-serving-cert" not found Oct 31 15:15:00.519978 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.519960 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2df8a3-04d0-407f-a4e6-269f49073e0f-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.521831 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.521808 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-metrics-certs\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:00.576247 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:00.576179 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scvdw\" (UniqueName: \"kubernetes.io/projected/ac2df8a3-04d0-407f-a4e6-269f49073e0f-kube-api-access-scvdw\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:01.025592 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:01.025539 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:01.028219 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:01.028196 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls\") pod \"maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:01.225708 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:01.225676 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:01.605862 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:01.605835 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl"] Oct 31 15:15:01.607685 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:15:01.607658 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2df8a3_04d0_407f_a4e6_269f49073e0f.slice/crio-bc48faf0f8f92cb502e78962e13e0b5c93550c58ce2231966099fc3dd1762461 WatchSource:0}: Error finding container bc48faf0f8f92cb502e78962e13e0b5c93550c58ce2231966099fc3dd1762461: Status 404 returned error can't find the container with id bc48faf0f8f92cb502e78962e13e0b5c93550c58ce2231966099fc3dd1762461 Oct 31 15:15:01.718719 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:01.718659 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" event={"ID":"ac2df8a3-04d0-407f-a4e6-269f49073e0f","Type":"ContainerStarted","Data":"bc48faf0f8f92cb502e78962e13e0b5c93550c58ce2231966099fc3dd1762461"} Oct 31 15:15:02.726864 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:02.726824 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" event={"ID":"ac2df8a3-04d0-407f-a4e6-269f49073e0f","Type":"ContainerStarted","Data":"854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47"} Oct 31 15:15:02.727283 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:02.727018 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:02.846728 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:02.846590 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" podStartSLOduration=1.907359503 podStartE2EDuration="2.846570029s" totalImagesPullingTime="939.210526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:02.843099195 +0000 UTC m=+1746.054340621" watchObservedRunningTime="2026-10-31 15:15:02.846570029 +0000 UTC m=+1746.057811442" Oct 31 15:15:08.738165 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:08.738091 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:15:11.029800 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.029766 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4"] Oct 31 15:15:11.035031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.035010 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:11.135029 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.135001 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2196e4\"" Oct 31 15:15:11.136659 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.136630 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2196e4-dockercfg-wpj7t\"" Oct 31 15:15:11.140632 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.140606 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4"] Oct 31 15:15:11.159225 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.159184 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwhzp\" (UniqueName: \"kubernetes.io/projected/c468ddf0-2371-4533-8df6-de0582394aa6-kube-api-access-jwhzp\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4\" (UID: \"c468ddf0-2371-4533-8df6-de0582394aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:11.159375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.159251 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c468ddf0-2371-4533-8df6-de0582394aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4\" (UID: \"c468ddf0-2371-4533-8df6-de0582394aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:11.260511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.260471 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhzp\" (UniqueName: \"kubernetes.io/projected/c468ddf0-2371-4533-8df6-de0582394aa6-kube-api-access-jwhzp\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4\" (UID: \"c468ddf0-2371-4533-8df6-de0582394aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:11.260723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.260541 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c468ddf0-2371-4533-8df6-de0582394aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4\" (UID: \"c468ddf0-2371-4533-8df6-de0582394aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:11.261348 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.261308 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c468ddf0-2371-4533-8df6-de0582394aa6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4\" (UID: \"c468ddf0-2371-4533-8df6-de0582394aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:11.319924 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.319851 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwhzp\" (UniqueName: \"kubernetes.io/projected/c468ddf0-2371-4533-8df6-de0582394aa6-kube-api-access-jwhzp\") pod \"payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4\" (UID: \"c468ddf0-2371-4533-8df6-de0582394aa6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:11.352663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.352636 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:11.731342 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.731314 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4"] Oct 31 15:15:11.733599 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:15:11.733573 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc468ddf0_2371_4533_8df6_de0582394aa6.slice/crio-98eb6d95c045bddf4da2357b87f4751f3a31f5e0469946b800ad9dfb84414a1e WatchSource:0}: Error finding container 98eb6d95c045bddf4da2357b87f4751f3a31f5e0469946b800ad9dfb84414a1e: Status 404 returned error can't find the container with id 98eb6d95c045bddf4da2357b87f4751f3a31f5e0469946b800ad9dfb84414a1e Oct 31 15:15:11.782401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:11.782372 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" event={"ID":"c468ddf0-2371-4533-8df6-de0582394aa6","Type":"ContainerStarted","Data":"98eb6d95c045bddf4da2357b87f4751f3a31f5e0469946b800ad9dfb84414a1e"} Oct 31 15:15:12.791080 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:12.791043 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" event={"ID":"c468ddf0-2371-4533-8df6-de0582394aa6","Type":"ContainerStarted","Data":"0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c"} Oct 31 15:15:12.791488 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:12.791178 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:12.903534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:12.903241 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" podStartSLOduration=2.903224143 podStartE2EDuration="2.903224143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:12.90295338 +0000 UTC m=+1756.114194791" watchObservedRunningTime="2026-10-31 15:15:12.903224143 +0000 UTC m=+1756.114465559" Oct 31 15:15:23.800261 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:23.800231 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:15:23.843995 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:23.843960 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88"] Oct 31 15:15:23.849485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:23.849457 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:23.901112 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:23.901075 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a538f1-serving-cert\"" Oct 31 15:15:23.903580 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:23.903545 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88"] Oct 31 15:15:23.948063 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:23.948024 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-metrics-certs\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:23.948268 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:23.948101 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l77b\" (UniqueName: \"kubernetes.io/projected/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-kube-api-access-4l77b\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:23.948368 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:23.948338 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:23.948486 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:23.948464 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.049684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.049645 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-metrics-certs\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.049874 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.049751 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l77b\" (UniqueName: \"kubernetes.io/projected/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-kube-api-access-4l77b\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.049923 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.049875 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.049964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.049953 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.050130 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:24.050088 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-serving-cert: secret "maas-api-e2e-ipp-b-a538f1-serving-cert" not found Oct 31 15:15:24.050254 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:24.050195 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls podName:5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb nodeName:}" failed. No retries permitted until 2026-10-31 15:15:24.550173192 +0000 UTC m=+1767.761414583 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls") pod "maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" (UID: "5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb") : secret "maas-api-e2e-ipp-b-a538f1-serving-cert" not found Oct 31 15:15:24.050748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.050679 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.052491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.052473 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-metrics-certs\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.111109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.110947 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l77b\" (UniqueName: \"kubernetes.io/projected/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-kube-api-access-4l77b\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.559306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.559270 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.561712 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.561672 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:24.767587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:24.767550 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:25.151482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:25.151458 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88"] Oct 31 15:15:25.154237 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:15:25.154206 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e55f7d9_6d54_485c_b8f5_f3b0a18c06cb.slice/crio-a66a6f0c8e7b2897c64692b8bb76d810725aa5ddf8fa51ba0b39774bdd32a732 WatchSource:0}: Error finding container a66a6f0c8e7b2897c64692b8bb76d810725aa5ddf8fa51ba0b39774bdd32a732: Status 404 returned error can't find the container with id a66a6f0c8e7b2897c64692b8bb76d810725aa5ddf8fa51ba0b39774bdd32a732 Oct 31 15:15:25.898509 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:25.898464 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" event={"ID":"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb","Type":"ContainerStarted","Data":"a66a6f0c8e7b2897c64692b8bb76d810725aa5ddf8fa51ba0b39774bdd32a732"} Oct 31 15:15:26.207903 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:26.207668 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:15:26.907132 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:26.907097 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" event={"ID":"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb","Type":"ContainerStarted","Data":"615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f"} Oct 31 15:15:26.907332 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:26.907174 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:26.908850 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:26.908825 4399 generic.go:356] "Generic (PLEG): container finished" podID="ea262732-121b-484c-bcc7-cfe929b2147f" containerID="25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51" exitCode=0 Oct 31 15:15:26.908950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:26.908902 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" event={"ID":"ea262732-121b-484c-bcc7-cfe929b2147f","Type":"ContainerDied","Data":"25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51"} Oct 31 15:15:26.909250 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:26.909236 4399 scope.go:122] "RemoveContainer" containerID="25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51" Oct 31 15:15:26.910650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:26.910619 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:15:26.910769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:26.910692 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:15:26.910769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:26.910750 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:15:27.023534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:27.023468 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" podStartSLOduration=3.139538248 podStartE2EDuration="4.023448385s" totalImagesPullingTime="883.910137ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:27.02002903 +0000 UTC m=+1770.231270444" watchObservedRunningTime="2026-10-31 15:15:27.023448385 +0000 UTC m=+1770.234689798" Oct 31 15:15:27.921495 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:27.921455 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" event={"ID":"ea262732-121b-484c-bcc7-cfe929b2147f","Type":"ContainerStarted","Data":"ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c"} Oct 31 15:15:28.534912 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:28.534877 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr"] Oct 31 15:15:28.535114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:28.535069 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" containerID="cri-o://66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b" gracePeriod=30 Oct 31 15:15:28.816039 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:28.815957 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:15:28.817090 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:28.817066 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 15:15:28.817203 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:28.817121 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 15:15:29.816451 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:29.816408 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 15:15:29.816892 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:29.816471 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 15:15:30.816519 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:30.816481 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 15:15:30.816984 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:30.816543 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 15:15:31.816935 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:31.816903 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 15:15:31.817337 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:31.816958 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 15:15:32.816750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:32.816680 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 31 15:15:32.816962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:32.816782 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 31 15:15:32.935446 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:32.935416 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:15:33.194105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.194025 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:15:33.298915 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.298875 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:15:33.299098 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.298950 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:15:33.808040 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.808015 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:15:33.821086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.821027 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:15:33.906706 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.906669 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-kube-api-access-sc498\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-kube-api-access-sc498\") pod \"6ded4067-f432-4509-824e-a862a94f24b3\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " Oct 31 15:15:33.906873 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.906769 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-certs\") pod \"6ded4067-f432-4509-824e-a862a94f24b3\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " Oct 31 15:15:33.906873 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.906816 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-data\") pod \"6ded4067-f432-4509-824e-a862a94f24b3\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " Oct 31 15:15:33.906873 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.906848 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ded4067-f432-4509-824e-a862a94f24b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ded4067-f432-4509-824e-a862a94f24b3-istiod-ca-cert\") pod \"6ded4067-f432-4509-824e-a862a94f24b3\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " Oct 31 15:15:33.907034 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.906896 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-credential-socket\") pod \"6ded4067-f432-4509-824e-a862a94f24b3\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " Oct 31 15:15:33.907034 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.906938 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-socket\") pod \"6ded4067-f432-4509-824e-a862a94f24b3\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " Oct 31 15:15:33.907034 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907014 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-istio-token\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-istio-token\") pod \"6ded4067-f432-4509-824e-a862a94f24b3\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " Oct 31 15:15:33.907187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907070 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-envoy\") pod \"6ded4067-f432-4509-824e-a862a94f24b3\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " Oct 31 15:15:33.907187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907116 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-certs" pod "6ded4067-f432-4509-824e-a862a94f24b3" (UID: "6ded4067-f432-4509-824e-a862a94f24b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:15:33.907187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907133 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ded4067-f432-4509-824e-a862a94f24b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ded4067-f432-4509-824e-a862a94f24b3-istio-podinfo\") pod \"6ded4067-f432-4509-824e-a862a94f24b3\" (UID: \"6ded4067-f432-4509-824e-a862a94f24b3\") " Oct 31 15:15:33.907187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907131 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-data" pod "6ded4067-f432-4509-824e-a862a94f24b3" (UID: "6ded4067-f432-4509-824e-a862a94f24b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:15:33.907187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907173 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-credential-socket" pod "6ded4067-f432-4509-824e-a862a94f24b3" (UID: "6ded4067-f432-4509-824e-a862a94f24b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:15:33.907438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907212 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-socket" pod "6ded4067-f432-4509-824e-a862a94f24b3" (UID: "6ded4067-f432-4509-824e-a862a94f24b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:15:33.907438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907276 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ded4067-f432-4509-824e-a862a94f24b3-istiod-ca-cert" pod "6ded4067-f432-4509-824e-a862a94f24b3" (UID: "6ded4067-f432-4509-824e-a862a94f24b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:15:33.907837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907794 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:33.908040 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.907821 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:33.908126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.908050 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ded4067-f432-4509-824e-a862a94f24b3-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:33.908126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.908065 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:33.908126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.908078 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:33.909247 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.909222 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-envoy" pod "6ded4067-f432-4509-824e-a862a94f24b3" (UID: "6ded4067-f432-4509-824e-a862a94f24b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:15:33.909338 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.909228 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-kube-api-access-sc498" pod "6ded4067-f432-4509-824e-a862a94f24b3" (UID: "6ded4067-f432-4509-824e-a862a94f24b3"). InnerVolumeSpecName "kube-api-access-sc498". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:33.909338 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.909301 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-istio-token" pod "6ded4067-f432-4509-824e-a862a94f24b3" (UID: "6ded4067-f432-4509-824e-a862a94f24b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:33.909429 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.909410 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ded4067-f432-4509-824e-a862a94f24b3-istio-podinfo" pod "6ded4067-f432-4509-824e-a862a94f24b3" (UID: "6ded4067-f432-4509-824e-a862a94f24b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:15:33.982179 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.982150 4399 generic.go:356] "Generic (PLEG): container finished" podID="6ded4067-f432-4509-824e-a862a94f24b3" containerID="66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b" exitCode=0 Oct 31 15:15:33.982360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.982210 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" Oct 31 15:15:33.982360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.982242 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" event={"ID":"6ded4067-f432-4509-824e-a862a94f24b3","Type":"ContainerDied","Data":"66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b"} Oct 31 15:15:33.982360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.982298 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr" event={"ID":"6ded4067-f432-4509-824e-a862a94f24b3","Type":"ContainerDied","Data":"56816b21ca196976a6aed032ed15f1f0c7d39121a79446944b4b5d9e964a8ab8"} Oct 31 15:15:33.982360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.982321 4399 scope.go:122] "RemoveContainer" containerID="66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b" Oct 31 15:15:33.983616 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.983601 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:15:33.993731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.993688 4399 scope.go:122] "RemoveContainer" containerID="66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b" Oct 31 15:15:33.994005 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:33.993987 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b\": container with ID starting with 66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b not found: ID does not exist" containerID="66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b" Oct 31 15:15:33.994049 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:33.994017 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b"} err="failed to get container status \"66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b\": rpc error: code = NotFound desc = could not find container \"66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b\": container with ID starting with 66c6712e20d2f925839e9ed4ec5cecfe6a1d9a1b9e77c21b375e7674c93ee49b not found: ID does not exist" Oct 31 15:15:34.009865 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:34.009844 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc498\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-kube-api-access-sc498\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:34.009865 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:34.009868 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ded4067-f432-4509-824e-a862a94f24b3-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:34.009865 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:34.009877 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ded4067-f432-4509-824e-a862a94f24b3-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:34.010084 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:34.009886 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ded4067-f432-4509-824e-a862a94f24b3-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:34.274175 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:34.274143 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr"] Oct 31 15:15:34.278475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:34.278453 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-xrrgr"] Oct 31 15:15:34.898608 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:34.898579 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ded4067-f432-4509-824e-a862a94f24b3" path="/var/lib/kubelet/pods/6ded4067-f432-4509-824e-a862a94f24b3/volumes" Oct 31 15:15:35.472031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.471998 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v"] Oct 31 15:15:35.472616 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.472603 4399 cpu_manager.go:414] "containerMap: removing container" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" Oct 31 15:15:35.472665 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.472618 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" Oct 31 15:15:35.472772 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.472762 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ded4067-f432-4509-824e-a862a94f24b3" containerName="istio-proxy" Oct 31 15:15:35.477583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.477566 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.533231 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.533206 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v"] Oct 31 15:15:35.578969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.578930 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f76cce-serving-cert\"" Oct 31 15:15:35.631276 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.631238 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv7pw\" (UniqueName: \"kubernetes.io/projected/16c062f9-11cb-494d-ac86-bd88848eb94e-kube-api-access-jv7pw\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.631444 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.631326 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.631577 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.631557 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-metrics-certs\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.631650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.631609 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c062f9-11cb-494d-ac86-bd88848eb94e-openshift-service-ca\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.734235 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.734129 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-metrics-certs\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.734235 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.734200 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c062f9-11cb-494d-ac86-bd88848eb94e-openshift-service-ca\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.734485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.734398 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv7pw\" (UniqueName: \"kubernetes.io/projected/16c062f9-11cb-494d-ac86-bd88848eb94e-kube-api-access-jv7pw\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.734485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.734449 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.734726 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:35.734656 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-serving-cert: secret "maas-api-e2e-api-a-f76cce-serving-cert" not found Oct 31 15:15:35.734812 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:35.734751 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls podName:16c062f9-11cb-494d-ac86-bd88848eb94e nodeName:}" failed. No retries permitted until 2026-10-31 15:15:36.234730861 +0000 UTC m=+1779.445972271 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls") pod "maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" (UID: "16c062f9-11cb-494d-ac86-bd88848eb94e") : secret "maas-api-e2e-api-a-f76cce-serving-cert" not found Oct 31 15:15:35.735557 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.735526 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c062f9-11cb-494d-ac86-bd88848eb94e-openshift-service-ca\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.739357 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.739334 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-metrics-certs\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:35.791866 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:35.791833 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv7pw\" (UniqueName: \"kubernetes.io/projected/16c062f9-11cb-494d-ac86-bd88848eb94e-kube-api-access-jv7pw\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:36.241953 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:36.241908 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:36.244738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:36.244686 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls\") pod \"maas-api-e2e-api-a-f76cce-8548978f9b-97q6v\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:36.391378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:36.391348 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:36.769567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:36.769538 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v"] Oct 31 15:15:36.771292 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:15:36.771261 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16c062f9_11cb_494d_ac86_bd88848eb94e.slice/crio-9a26a8ac3d14a63c7d1b5663386d8bf800cd9639dc2005951746bc827d461994 WatchSource:0}: Error finding container 9a26a8ac3d14a63c7d1b5663386d8bf800cd9639dc2005951746bc827d461994: Status 404 returned error can't find the container with id 9a26a8ac3d14a63c7d1b5663386d8bf800cd9639dc2005951746bc827d461994 Oct 31 15:15:37.004965 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:37.004925 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" event={"ID":"16c062f9-11cb-494d-ac86-bd88848eb94e","Type":"ContainerStarted","Data":"9a26a8ac3d14a63c7d1b5663386d8bf800cd9639dc2005951746bc827d461994"} Oct 31 15:15:38.013543 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:38.013505 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" event={"ID":"16c062f9-11cb-494d-ac86-bd88848eb94e","Type":"ContainerStarted","Data":"4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718"} Oct 31 15:15:38.014096 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:38.013570 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:38.130896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:38.130847 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" podStartSLOduration=2.261345347 podStartE2EDuration="3.130830013s" totalImagesPullingTime="869.484666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:15:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:15:38.127259331 +0000 UTC m=+1781.338500744" watchObservedRunningTime="2026-10-31 15:15:38.130830013 +0000 UTC m=+1781.342071426" Oct 31 15:15:44.026435 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:44.026400 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:15:54.094782 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.094744 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d"] Oct 31 15:15:54.095204 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.094952 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" podUID="db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" containerName="maas-api" containerID="cri-o://ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9" gracePeriod=30 Oct 31 15:15:54.377634 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.377608 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:15:54.470981 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.470939 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-metrics-certs\") pod \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " Oct 31 15:15:54.471292 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.471272 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-openshift-service-ca\") pod \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " Oct 31 15:15:54.471950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.471468 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls\") pod \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " Oct 31 15:15:54.471950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.471590 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-openshift-service-ca" pod "db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" (UID: "db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:15:54.471950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.471618 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-kube-api-access-7k7bh\" (UniqueName: \"kubernetes.io/projected/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-kube-api-access-7k7bh\") pod \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\" (UID: \"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e\") " Oct 31 15:15:54.472455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.472435 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:54.474090 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.474059 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-metrics-certs" pod "db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" (UID: "db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:15:54.474495 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.474476 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-kube-api-access-7k7bh" pod "db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" (UID: "db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e"). InnerVolumeSpecName "kube-api-access-7k7bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:15:54.474635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.474620 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls" pod "db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" (UID: "db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:15:54.573472 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.573439 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:54.573472 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.573471 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k7bh\" (UniqueName: \"kubernetes.io/projected/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-kube-api-access-7k7bh\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:54.573472 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:54.573481 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:15:55.142523 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:55.142492 4399 generic.go:356] "Generic (PLEG): container finished" podID="db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" containerID="ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9" exitCode=0 Oct 31 15:15:55.142987 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:55.142538 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" Oct 31 15:15:55.142987 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:55.142568 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" event={"ID":"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e","Type":"ContainerDied","Data":"ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9"} Oct 31 15:15:55.142987 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:55.142626 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d" event={"ID":"db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e","Type":"ContainerDied","Data":"60aed8478ed133e618859baad0e6be3177b6f9f684680c4f4a29d3fa2519e82d"} Oct 31 15:15:55.142987 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:55.142653 4399 scope.go:122] "RemoveContainer" containerID="ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9" Oct 31 15:15:55.155085 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:55.155065 4399 scope.go:122] "RemoveContainer" containerID="ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9" Oct 31 15:15:55.155593 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:55.155567 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9\": container with ID starting with ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9 not found: ID does not exist" containerID="ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9" Oct 31 15:15:55.155667 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:55.155595 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9"} err="failed to get container status \"ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9\": rpc error: code = NotFound desc = could not find container \"ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9\": container with ID starting with ef68147e4155733c010b9e8455bf272f4ce9aa1128b2296c8216d2415c32b4c9 not found: ID does not exist" Oct 31 15:15:55.306573 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:55.306543 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d"] Oct 31 15:15:55.355795 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:55.355764 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f7251e-9996fb5ff-tg52d"] Oct 31 15:15:56.207323 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:15:56.207292 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:15:56.903206 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:15:56.903167 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" path="/var/lib/kubelet/pods/db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e/volumes" Oct 31 15:16:14.680594 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.680552 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr"] Oct 31 15:16:14.681608 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.681584 4399 cpu_manager.go:414] "containerMap: removing container" podUID="db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" containerName="maas-api" Oct 31 15:16:14.681759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.681611 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" containerName="maas-api" Oct 31 15:16:14.681830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.681807 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0f2ad7-3c56-4fb4-81bb-5a7bbbe9f73e" containerName="maas-api" Oct 31 15:16:14.701976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.687523 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:14.718471 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.718329 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr"] Oct 31 15:16:14.755323 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.755288 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv6m2\" (UniqueName: \"kubernetes.io/projected/e448a9d5-75a8-488a-bb87-3108abf4ac15-kube-api-access-lv6m2\") pod \"payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr\" (UID: \"e448a9d5-75a8-488a-bb87-3108abf4ac15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:14.755323 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.755322 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e448a9d5-75a8-488a-bb87-3108abf4ac15-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr\" (UID: \"e448a9d5-75a8-488a-bb87-3108abf4ac15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:14.761384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.761360 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-9ab5df\"" Oct 31 15:16:14.761384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.761368 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-9ab5df-dockercfg-l8k2l\"" Oct 31 15:16:14.856963 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.856779 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv6m2\" (UniqueName: \"kubernetes.io/projected/e448a9d5-75a8-488a-bb87-3108abf4ac15-kube-api-access-lv6m2\") pod \"payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr\" (UID: \"e448a9d5-75a8-488a-bb87-3108abf4ac15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:14.856963 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.856831 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e448a9d5-75a8-488a-bb87-3108abf4ac15-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr\" (UID: \"e448a9d5-75a8-488a-bb87-3108abf4ac15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:14.858168 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.858134 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e448a9d5-75a8-488a-bb87-3108abf4ac15-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr\" (UID: \"e448a9d5-75a8-488a-bb87-3108abf4ac15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:14.915005 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:14.914973 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv6m2\" (UniqueName: \"kubernetes.io/projected/e448a9d5-75a8-488a-bb87-3108abf4ac15-kube-api-access-lv6m2\") pod \"payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr\" (UID: \"e448a9d5-75a8-488a-bb87-3108abf4ac15\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:15.021576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:15.021542 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:15.404881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:15.404846 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr"] Oct 31 15:16:15.406264 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:16:15.406235 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode448a9d5_75a8_488a_bb87_3108abf4ac15.slice/crio-175e6e15470a9955a072e8c621be08d7da2917cc3d046259f2353d912a531011 WatchSource:0}: Error finding container 175e6e15470a9955a072e8c621be08d7da2917cc3d046259f2353d912a531011: Status 404 returned error can't find the container with id 175e6e15470a9955a072e8c621be08d7da2917cc3d046259f2353d912a531011 Oct 31 15:16:16.289096 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:16.289052 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" event={"ID":"e448a9d5-75a8-488a-bb87-3108abf4ac15","Type":"ContainerStarted","Data":"b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463"} Oct 31 15:16:16.289482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:16.289111 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" event={"ID":"e448a9d5-75a8-488a-bb87-3108abf4ac15","Type":"ContainerStarted","Data":"175e6e15470a9955a072e8c621be08d7da2917cc3d046259f2353d912a531011"} Oct 31 15:16:16.289482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:16.289138 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:16.403594 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:16.403542 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" podStartSLOduration=2.403528333 podStartE2EDuration="2.403528333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:16:16.401583999 +0000 UTC m=+1819.612825413" watchObservedRunningTime="2026-10-31 15:16:16.403528333 +0000 UTC m=+1819.614769744" Oct 31 15:16:26.207662 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:16:26.207622 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:16:27.298634 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:27.298601 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:16:56.207380 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:16:56.207343 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:16:58.352287 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.352255 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z"] Oct 31 15:16:58.356555 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.356536 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:16:58.420542 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.420492 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z"] Oct 31 15:16:58.502105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.502051 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bjz9\" (UniqueName: \"kubernetes.io/projected/a2ee0e99-7f27-4287-8253-3cb74a89f626-kube-api-access-7bjz9\") pod \"payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z\" (UID: \"a2ee0e99-7f27-4287-8253-3cb74a89f626\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:16:58.502449 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.502422 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee0e99-7f27-4287-8253-3cb74a89f626-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z\" (UID: \"a2ee0e99-7f27-4287-8253-3cb74a89f626\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:16:58.507162 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.507132 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9cc557\"" Oct 31 15:16:58.507524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.507493 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9cc557-dockercfg-sqqsk\"" Oct 31 15:16:58.604086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.604009 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bjz9\" (UniqueName: \"kubernetes.io/projected/a2ee0e99-7f27-4287-8253-3cb74a89f626-kube-api-access-7bjz9\") pod \"payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z\" (UID: \"a2ee0e99-7f27-4287-8253-3cb74a89f626\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:16:58.604086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.604082 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee0e99-7f27-4287-8253-3cb74a89f626-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z\" (UID: \"a2ee0e99-7f27-4287-8253-3cb74a89f626\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:16:58.604811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.604790 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee0e99-7f27-4287-8253-3cb74a89f626-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z\" (UID: \"a2ee0e99-7f27-4287-8253-3cb74a89f626\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:16:58.660889 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.660864 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bjz9\" (UniqueName: \"kubernetes.io/projected/a2ee0e99-7f27-4287-8253-3cb74a89f626-kube-api-access-7bjz9\") pod \"payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z\" (UID: \"a2ee0e99-7f27-4287-8253-3cb74a89f626\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:16:58.672103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:58.672079 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:16:59.050172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:59.050149 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z"] Oct 31 15:16:59.052597 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:16:59.052563 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ee0e99_7f27_4287_8253_3cb74a89f626.slice/crio-4be5acfc383d1bbd2c7060d1b423ddefa14be4767f938703143b43c0e43b6dac WatchSource:0}: Error finding container 4be5acfc383d1bbd2c7060d1b423ddefa14be4767f938703143b43c0e43b6dac: Status 404 returned error can't find the container with id 4be5acfc383d1bbd2c7060d1b423ddefa14be4767f938703143b43c0e43b6dac Oct 31 15:16:59.556442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:59.556412 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" event={"ID":"a2ee0e99-7f27-4287-8253-3cb74a89f626","Type":"ContainerStarted","Data":"5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6"} Oct 31 15:16:59.556831 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:59.556463 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" event={"ID":"a2ee0e99-7f27-4287-8253-3cb74a89f626","Type":"ContainerStarted","Data":"4be5acfc383d1bbd2c7060d1b423ddefa14be4767f938703143b43c0e43b6dac"} Oct 31 15:16:59.556831 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:59.556487 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:16:59.671921 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:16:59.671872 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" podStartSLOduration=1.6718577749999999 podStartE2EDuration="1.671857775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:16:59.667981828 +0000 UTC m=+1862.879223273" watchObservedRunningTime="2026-10-31 15:16:59.671857775 +0000 UTC m=+1862.883099186" Oct 31 15:17:10.567635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:10.567602 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:17:11.536990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:11.536963 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kzfcq_5aac9441-7f91-49f3-9472-8fcd34fec615/payload-processing/0.log" Oct 31 15:17:11.781193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:11.781168 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kzfcq_5aac9441-7f91-49f3-9472-8fcd34fec615/payload-processing/0.log" Oct 31 15:17:17.373709 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.373657 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f"] Oct 31 15:17:17.378638 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.378615 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.434573 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.434544 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f"] Oct 31 15:17:17.479961 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.479933 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-9ab5df-serving-cert\"" Oct 31 15:17:17.550585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.550552 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-metrics-certs\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.550745 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.550598 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57s7z\" (UniqueName: \"kubernetes.io/projected/0d406da3-3311-47a8-b443-679c059505a6-kube-api-access-57s7z\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.550827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.550806 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d406da3-3311-47a8-b443-679c059505a6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.550990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.550976 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.652143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.652047 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d406da3-3311-47a8-b443-679c059505a6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.652320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.652164 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.652320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.652194 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-metrics-certs\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.652320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.652231 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57s7z\" (UniqueName: \"kubernetes.io/projected/0d406da3-3311-47a8-b443-679c059505a6-kube-api-access-57s7z\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.652512 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:17:17.652332 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-serving-cert: secret "maas-api-e2e-praxis-a-9ab5df-serving-cert" not found Oct 31 15:17:17.652512 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:17:17.652408 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls podName:0d406da3-3311-47a8-b443-679c059505a6 nodeName:}" failed. No retries permitted until 2026-10-31 15:17:18.152388281 +0000 UTC m=+1881.363629672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls") pod "maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" (UID: "0d406da3-3311-47a8-b443-679c059505a6") : secret "maas-api-e2e-praxis-a-9ab5df-serving-cert" not found Oct 31 15:17:17.652859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.652836 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d406da3-3311-47a8-b443-679c059505a6-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.654789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.654770 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-metrics-certs\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:17.709808 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:17.709774 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57s7z\" (UniqueName: \"kubernetes.io/projected/0d406da3-3311-47a8-b443-679c059505a6-kube-api-access-57s7z\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:18.158784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:18.158756 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:18.161429 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:18.161403 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls\") pod \"maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:18.294982 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:18.294952 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:18.681731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:18.681672 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f"] Oct 31 15:17:18.683498 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:17:18.683459 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d406da3_3311_47a8_b443_679c059505a6.slice/crio-09ce6ed2e5e448cf185f96f50035c33a544c6483ca7baf070bb13552e148d474 WatchSource:0}: Error finding container 09ce6ed2e5e448cf185f96f50035c33a544c6483ca7baf070bb13552e148d474: Status 404 returned error can't find the container with id 09ce6ed2e5e448cf185f96f50035c33a544c6483ca7baf070bb13552e148d474 Oct 31 15:17:18.685076 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:18.685053 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 15:17:18.690920 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:18.690895 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" event={"ID":"0d406da3-3311-47a8-b443-679c059505a6","Type":"ContainerStarted","Data":"09ce6ed2e5e448cf185f96f50035c33a544c6483ca7baf070bb13552e148d474"} Oct 31 15:17:19.698756 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:19.698691 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" event={"ID":"0d406da3-3311-47a8-b443-679c059505a6","Type":"ContainerStarted","Data":"4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2"} Oct 31 15:17:19.699077 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:19.698781 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:19.819979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:19.819925 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" podStartSLOduration=1.916589477 podStartE2EDuration="2.819912277s" totalImagesPullingTime="903.3228ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:17:19.817659282 +0000 UTC m=+1883.028900695" watchObservedRunningTime="2026-10-31 15:17:19.819912277 +0000 UTC m=+1883.031153689" Oct 31 15:17:21.737154 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:21.737122 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht"] Oct 31 15:17:21.737681 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:21.737309 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" podUID="e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" containerName="maas-api" containerID="cri-o://a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40" gracePeriod=30 Oct 31 15:17:21.788063 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:21.788028 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" podUID="e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.94:8443/health\": dial tcp 10.133.0.94:8443: connect: connection refused" Oct 31 15:17:22.056203 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.056173 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:17:22.114018 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.113980 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-openshift-service-ca\") pod \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " Oct 31 15:17:22.114222 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.114205 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-kube-api-access-vlw6c\" (UniqueName: \"kubernetes.io/projected/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-kube-api-access-vlw6c\") pod \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " Oct 31 15:17:22.114335 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.114316 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls\") pod \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " Oct 31 15:17:22.114402 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.114370 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-metrics-certs\") pod \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\" (UID: \"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5\") " Oct 31 15:17:22.114457 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.114408 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-openshift-service-ca" pod "e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" (UID: "e2a4c047-3ed2-4ad3-89b1-6349b35fbad5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:17:22.114908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.114881 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:22.117257 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.117232 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-kube-api-access-vlw6c" pod "e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" (UID: "e2a4c047-3ed2-4ad3-89b1-6349b35fbad5"). InnerVolumeSpecName "kube-api-access-vlw6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:22.117590 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.117551 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-metrics-certs" pod "e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" (UID: "e2a4c047-3ed2-4ad3-89b1-6349b35fbad5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:17:22.120249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.120223 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls" pod "e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" (UID: "e2a4c047-3ed2-4ad3-89b1-6349b35fbad5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:17:22.216074 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.216038 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlw6c\" (UniqueName: \"kubernetes.io/projected/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-kube-api-access-vlw6c\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:22.216074 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.216070 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:22.216074 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.216081 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:22.244597 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.244559 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4"] Oct 31 15:17:22.244891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.244866 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" podUID="c468ddf0-2371-4533-8df6-de0582394aa6" containerName="payload-pre-processing" containerID="cri-o://0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c" gracePeriod=30 Oct 31 15:17:22.495004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.494980 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:17:22.520052 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.520025 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c468ddf0-2371-4533-8df6-de0582394aa6-kube-api-access-jwhzp\" (UniqueName: \"kubernetes.io/projected/c468ddf0-2371-4533-8df6-de0582394aa6-kube-api-access-jwhzp\") pod \"c468ddf0-2371-4533-8df6-de0582394aa6\" (UID: \"c468ddf0-2371-4533-8df6-de0582394aa6\") " Oct 31 15:17:22.520196 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.520102 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c468ddf0-2371-4533-8df6-de0582394aa6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c468ddf0-2371-4533-8df6-de0582394aa6-plugins-config-volume\") pod \"c468ddf0-2371-4533-8df6-de0582394aa6\" (UID: \"c468ddf0-2371-4533-8df6-de0582394aa6\") " Oct 31 15:17:22.520532 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.520501 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c468ddf0-2371-4533-8df6-de0582394aa6-plugins-config-volume" pod "c468ddf0-2371-4533-8df6-de0582394aa6" (UID: "c468ddf0-2371-4533-8df6-de0582394aa6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:17:22.522250 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.522222 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c468ddf0-2371-4533-8df6-de0582394aa6-kube-api-access-jwhzp" pod "c468ddf0-2371-4533-8df6-de0582394aa6" (UID: "c468ddf0-2371-4533-8df6-de0582394aa6"). InnerVolumeSpecName "kube-api-access-jwhzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:22.621954 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.621877 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwhzp\" (UniqueName: \"kubernetes.io/projected/c468ddf0-2371-4533-8df6-de0582394aa6-kube-api-access-jwhzp\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:22.621954 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.621907 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c468ddf0-2371-4533-8df6-de0582394aa6-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:22.778047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.777532 4399 generic.go:356] "Generic (PLEG): container finished" podID="c468ddf0-2371-4533-8df6-de0582394aa6" containerID="0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c" exitCode=0 Oct 31 15:17:22.778047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.777656 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" event={"ID":"c468ddf0-2371-4533-8df6-de0582394aa6","Type":"ContainerDied","Data":"0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c"} Oct 31 15:17:22.778047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.777727 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" event={"ID":"c468ddf0-2371-4533-8df6-de0582394aa6","Type":"ContainerDied","Data":"98eb6d95c045bddf4da2357b87f4751f3a31f5e0469946b800ad9dfb84414a1e"} Oct 31 15:17:22.778047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.777764 4399 scope.go:122] "RemoveContainer" containerID="0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c" Oct 31 15:17:22.778047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.777941 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4" Oct 31 15:17:22.783845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.783819 4399 generic.go:356] "Generic (PLEG): container finished" podID="e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" containerID="a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40" exitCode=0 Oct 31 15:17:22.784015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.783995 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" event={"ID":"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5","Type":"ContainerDied","Data":"a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40"} Oct 31 15:17:22.784120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.784034 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" event={"ID":"e2a4c047-3ed2-4ad3-89b1-6349b35fbad5","Type":"ContainerDied","Data":"d3c65c569de23d9c811f5286cded3d8ff0726e94f510ed4ed309798c01416550"} Oct 31 15:17:22.784120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.784093 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht" Oct 31 15:17:22.795985 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.795966 4399 scope.go:122] "RemoveContainer" containerID="0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c" Oct 31 15:17:22.796375 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:17:22.796347 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c\": container with ID starting with 0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c not found: ID does not exist" containerID="0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c" Oct 31 15:17:22.796453 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.796391 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c"} err="failed to get container status \"0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c\": rpc error: code = NotFound desc = could not find container \"0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c\": container with ID starting with 0cab673883646b5ea02d6879aca6d8328d3c1c859178f84ce0d2bb392df1f06c not found: ID does not exist" Oct 31 15:17:22.796453 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.796417 4399 scope.go:122] "RemoveContainer" containerID="a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40" Oct 31 15:17:22.806508 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.806491 4399 scope.go:122] "RemoveContainer" containerID="a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40" Oct 31 15:17:22.806771 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:17:22.806745 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40\": container with ID starting with a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40 not found: ID does not exist" containerID="a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40" Oct 31 15:17:22.806865 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.806777 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40"} err="failed to get container status \"a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40\": rpc error: code = NotFound desc = could not find container \"a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40\": container with ID starting with a024ce1f17b49b2a5076c37d3df546d35497e82a587c286280e88a62ff396a40 not found: ID does not exist" Oct 31 15:17:22.945269 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.945230 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht"] Oct 31 15:17:22.948923 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:22.948898 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2196e4-cc84bd84b-56lht"] Oct 31 15:17:23.056249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.056218 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4"] Oct 31 15:17:23.061967 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.061945 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-2196e4-5d6b5f7d87-89mz4"] Oct 31 15:17:23.303134 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.303067 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm"] Oct 31 15:17:23.303741 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.303725 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" containerName="maas-api" Oct 31 15:17:23.303798 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.303742 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" containerName="maas-api" Oct 31 15:17:23.303798 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.303781 4399 cpu_manager.go:414] "containerMap: removing container" podUID="c468ddf0-2371-4533-8df6-de0582394aa6" containerName="payload-pre-processing" Oct 31 15:17:23.303798 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.303786 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c468ddf0-2371-4533-8df6-de0582394aa6" containerName="payload-pre-processing" Oct 31 15:17:23.303902 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.303892 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c468ddf0-2371-4533-8df6-de0582394aa6" containerName="payload-pre-processing" Oct 31 15:17:23.303936 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.303911 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" containerName="maas-api" Oct 31 15:17:23.308828 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.308814 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.354223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.354195 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8969a7-0130-4522-8955-549f2e16ae15-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.354341 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.354234 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.354341 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.354322 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bv4m\" (UniqueName: \"kubernetes.io/projected/ae8969a7-0130-4522-8955-549f2e16ae15-kube-api-access-5bv4m\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.354417 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.354355 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.361223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.361195 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ad5bfe-serving-cert\"" Oct 31 15:17:23.364674 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.364651 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm"] Oct 31 15:17:23.455600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.455550 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8969a7-0130-4522-8955-549f2e16ae15-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.455793 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.455607 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.455793 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.455666 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bv4m\" (UniqueName: \"kubernetes.io/projected/ae8969a7-0130-4522-8955-549f2e16ae15-kube-api-access-5bv4m\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.455793 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.455721 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.455995 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:17:23.455828 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-serving-cert: secret "maas-api-e2e-ipp-b-ad5bfe-serving-cert" not found Oct 31 15:17:23.455995 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:17:23.455943 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls podName:ae8969a7-0130-4522-8955-549f2e16ae15 nodeName:}" failed. No retries permitted until 2026-10-31 15:17:23.955906273 +0000 UTC m=+1887.167147680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls") pod "maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" (UID: "ae8969a7-0130-4522-8955-549f2e16ae15") : secret "maas-api-e2e-ipp-b-ad5bfe-serving-cert" not found Oct 31 15:17:23.456453 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.456424 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8969a7-0130-4522-8955-549f2e16ae15-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.459024 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.459001 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-metrics-certs\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.515024 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.514999 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bv4m\" (UniqueName: \"kubernetes.io/projected/ae8969a7-0130-4522-8955-549f2e16ae15-kube-api-access-5bv4m\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.962617 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.962578 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:23.965357 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:23.965326 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:24.223618 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:24.223544 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:24.599602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:24.599574 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm"] Oct 31 15:17:24.600956 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:17:24.600931 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8969a7_0130_4522_8955_549f2e16ae15.slice/crio-ee258d3666caf26792ac65650ed0657126e2b6a1225682c1366dac4f44292014 WatchSource:0}: Error finding container ee258d3666caf26792ac65650ed0657126e2b6a1225682c1366dac4f44292014: Status 404 returned error can't find the container with id ee258d3666caf26792ac65650ed0657126e2b6a1225682c1366dac4f44292014 Oct 31 15:17:24.808488 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:24.808448 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" event={"ID":"ae8969a7-0130-4522-8955-549f2e16ae15","Type":"ContainerStarted","Data":"ee258d3666caf26792ac65650ed0657126e2b6a1225682c1366dac4f44292014"} Oct 31 15:17:24.896343 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:24.896262 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c468ddf0-2371-4533-8df6-de0582394aa6" path="/var/lib/kubelet/pods/c468ddf0-2371-4533-8df6-de0582394aa6/volumes" Oct 31 15:17:24.896731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:24.896716 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2a4c047-3ed2-4ad3-89b1-6349b35fbad5" path="/var/lib/kubelet/pods/e2a4c047-3ed2-4ad3-89b1-6349b35fbad5/volumes" Oct 31 15:17:25.710783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:25.710759 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:17:25.815556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:25.815524 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" event={"ID":"ae8969a7-0130-4522-8955-549f2e16ae15","Type":"ContainerStarted","Data":"36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18"} Oct 31 15:17:25.815755 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:25.815663 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:25.939811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:25.939754 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" podStartSLOduration=2.1080965210000002 podStartE2EDuration="2.939736027s" totalImagesPullingTime="831.639506ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:17:25.936832463 +0000 UTC m=+1889.148073885" watchObservedRunningTime="2026-10-31 15:17:25.939736027 +0000 UTC m=+1889.150977439" Oct 31 15:17:26.208726 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:17:26.208664 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:17:31.826791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:31.826762 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:17:39.215770 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.215741 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88"] Oct 31 15:17:39.216204 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.215952 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" podUID="5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" containerName="maas-api" containerID="cri-o://615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f" gracePeriod=30 Oct 31 15:17:39.508736 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.508683 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:17:39.563616 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.563584 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-metrics-certs\") pod \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " Oct 31 15:17:39.563814 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.563661 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls\") pod \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " Oct 31 15:17:39.563814 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.563711 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-openshift-service-ca\") pod \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " Oct 31 15:17:39.563954 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.563935 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-kube-api-access-4l77b\" (UniqueName: \"kubernetes.io/projected/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-kube-api-access-4l77b\") pod \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\" (UID: \"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb\") " Oct 31 15:17:39.564137 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.564108 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-openshift-service-ca" pod "5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" (UID: "5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:17:39.564442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.564427 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:39.565797 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.565770 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-metrics-certs" pod "5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" (UID: "5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:17:39.565891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.565879 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls" pod "5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" (UID: "5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:17:39.566008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.565991 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-kube-api-access-4l77b" pod "5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" (UID: "5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb"). InnerVolumeSpecName "kube-api-access-4l77b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:17:39.665360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.665323 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l77b\" (UniqueName: \"kubernetes.io/projected/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-kube-api-access-4l77b\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:39.665360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.665350 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:39.665360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.665360 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:17:39.933010 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.932934 4399 generic.go:356] "Generic (PLEG): container finished" podID="5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" containerID="615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f" exitCode=0 Oct 31 15:17:39.933010 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.932983 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" Oct 31 15:17:39.933192 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.933024 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" event={"ID":"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb","Type":"ContainerDied","Data":"615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f"} Oct 31 15:17:39.933192 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.933076 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88" event={"ID":"5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb","Type":"ContainerDied","Data":"a66a6f0c8e7b2897c64692b8bb76d810725aa5ddf8fa51ba0b39774bdd32a732"} Oct 31 15:17:39.933192 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.933101 4399 scope.go:122] "RemoveContainer" containerID="615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f" Oct 31 15:17:39.944349 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.944319 4399 scope.go:122] "RemoveContainer" containerID="615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f" Oct 31 15:17:39.944620 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:17:39.944602 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f\": container with ID starting with 615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f not found: ID does not exist" containerID="615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f" Oct 31 15:17:39.944663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:39.944629 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f"} err="failed to get container status \"615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f\": rpc error: code = NotFound desc = could not find container \"615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f\": container with ID starting with 615b43b0a27d36a97b53d8e20e66b58bdfdb3467c8a413ecba02244a6127858f not found: ID does not exist" Oct 31 15:17:40.097549 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:40.097520 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88"] Oct 31 15:17:40.101671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:40.101649 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a538f1-54d765f77d-mrv88"] Oct 31 15:17:40.896680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:17:40.896639 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" path="/var/lib/kubelet/pods/5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb/volumes" Oct 31 15:17:56.207401 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:17:56.207363 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:18:13.450591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:13.450503 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk"] Oct 31 15:18:13.451193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:13.450841 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" containerID="cri-o://dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b" gracePeriod=30 Oct 31 15:18:13.670578 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:13.670547 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:18:13.670766 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:13.670598 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:18:18.740491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.740464 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 15:18:18.770740 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.770692 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4907085c-e7cb-4cec-a720-4fa41cf6060c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4907085c-e7cb-4cec-a720-4fa41cf6060c-istiod-ca-cert\") pod \"4907085c-e7cb-4cec-a720-4fa41cf6060c\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " Oct 31 15:18:18.770857 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.770755 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-data\") pod \"4907085c-e7cb-4cec-a720-4fa41cf6060c\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " Oct 31 15:18:18.770857 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.770781 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-credential-socket\") pod \"4907085c-e7cb-4cec-a720-4fa41cf6060c\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " Oct 31 15:18:18.770857 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.770811 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-envoy\") pod \"4907085c-e7cb-4cec-a720-4fa41cf6060c\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " Oct 31 15:18:18.770857 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.770832 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-certs\") pod \"4907085c-e7cb-4cec-a720-4fa41cf6060c\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " Oct 31 15:18:18.771023 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.770913 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-podinfo\") pod \"4907085c-e7cb-4cec-a720-4fa41cf6060c\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " Oct 31 15:18:18.771023 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.770971 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-kube-api-access-x5lcc\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-kube-api-access-x5lcc\") pod \"4907085c-e7cb-4cec-a720-4fa41cf6060c\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " Oct 31 15:18:18.771128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.771033 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-socket\") pod \"4907085c-e7cb-4cec-a720-4fa41cf6060c\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " Oct 31 15:18:18.771185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.771088 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4907085c-e7cb-4cec-a720-4fa41cf6060c-istiod-ca-cert" pod "4907085c-e7cb-4cec-a720-4fa41cf6060c" (UID: "4907085c-e7cb-4cec-a720-4fa41cf6060c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:18:18.771185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.771088 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-credential-socket" pod "4907085c-e7cb-4cec-a720-4fa41cf6060c" (UID: "4907085c-e7cb-4cec-a720-4fa41cf6060c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:18.771185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.771139 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-token\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-token\") pod \"4907085c-e7cb-4cec-a720-4fa41cf6060c\" (UID: \"4907085c-e7cb-4cec-a720-4fa41cf6060c\") " Oct 31 15:18:18.771344 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.771200 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-certs" pod "4907085c-e7cb-4cec-a720-4fa41cf6060c" (UID: "4907085c-e7cb-4cec-a720-4fa41cf6060c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:18.771452 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.771427 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-socket" pod "4907085c-e7cb-4cec-a720-4fa41cf6060c" (UID: "4907085c-e7cb-4cec-a720-4fa41cf6060c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:18.771452 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.771438 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-data" pod "4907085c-e7cb-4cec-a720-4fa41cf6060c" (UID: "4907085c-e7cb-4cec-a720-4fa41cf6060c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:18.772060 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.772035 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:18.772158 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.772066 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4907085c-e7cb-4cec-a720-4fa41cf6060c-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:18.772158 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.772081 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:18.772158 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.772094 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:18.772158 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.772108 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:18.773227 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.773195 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-envoy" pod "4907085c-e7cb-4cec-a720-4fa41cf6060c" (UID: "4907085c-e7cb-4cec-a720-4fa41cf6060c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:18.773328 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.773313 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-kube-api-access-x5lcc" pod "4907085c-e7cb-4cec-a720-4fa41cf6060c" (UID: "4907085c-e7cb-4cec-a720-4fa41cf6060c"). InnerVolumeSpecName "kube-api-access-x5lcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:18.773782 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.773755 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-podinfo" pod "4907085c-e7cb-4cec-a720-4fa41cf6060c" (UID: "4907085c-e7cb-4cec-a720-4fa41cf6060c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:18:18.773782 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.773766 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-token" pod "4907085c-e7cb-4cec-a720-4fa41cf6060c" (UID: "4907085c-e7cb-4cec-a720-4fa41cf6060c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:18.881721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.874906 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:18.881721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.874939 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5lcc\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-kube-api-access-x5lcc\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:18.881721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.874955 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:18.881721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:18.874976 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4907085c-e7cb-4cec-a720-4fa41cf6060c-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:19.216964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:19.216935 4399 generic.go:356] "Generic (PLEG): container finished" podID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerID="dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b" exitCode=0 Oct 31 15:18:19.217136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:19.217006 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" Oct 31 15:18:19.217136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:19.217023 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" event={"ID":"4907085c-e7cb-4cec-a720-4fa41cf6060c","Type":"ContainerDied","Data":"dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b"} Oct 31 15:18:19.217136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:19.217082 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk" event={"ID":"4907085c-e7cb-4cec-a720-4fa41cf6060c","Type":"ContainerDied","Data":"c94d2c68342d8de0c283beb92768e3dab622faca4df82c512904ea5b2c0c35d7"} Oct 31 15:18:19.217136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:19.217111 4399 scope.go:122] "RemoveContainer" containerID="dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b" Oct 31 15:18:19.227058 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:19.227038 4399 scope.go:122] "RemoveContainer" containerID="dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b" Oct 31 15:18:19.227323 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:18:19.227301 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b\": container with ID starting with dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b not found: ID does not exist" containerID="dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b" Oct 31 15:18:19.227364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:19.227339 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b"} err="failed to get container status \"dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b\": rpc error: code = NotFound desc = could not find container \"dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b\": container with ID starting with dfdb85f80527ff487a7eb0d256ed273d3c300e89afe2df2ab3f4eb95f9c7b94b not found: ID does not exist" Oct 31 15:18:19.381406 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:19.381374 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk"] Oct 31 15:18:19.430922 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:19.430890 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-b4gbk"] Oct 31 15:18:20.897927 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:20.897895 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" path="/var/lib/kubelet/pods/4907085c-e7cb-4cec-a720-4fa41cf6060c/volumes" Oct 31 15:18:26.207508 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:18:26.207469 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:18:37.956974 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:37.956928 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw"] Oct 31 15:18:37.957475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:37.957176 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" podUID="b7cc0df2-9462-47b4-96f0-6107c0cdc28e" containerName="maas-api" containerID="cri-o://2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872" gracePeriod=30 Oct 31 15:18:38.087930 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.087899 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh"] Oct 31 15:18:38.088577 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.088561 4399 cpu_manager.go:414] "containerMap: removing container" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" Oct 31 15:18:38.088648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.088578 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" Oct 31 15:18:38.088648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.088600 4399 cpu_manager.go:414] "containerMap: removing container" podUID="5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" containerName="maas-api" Oct 31 15:18:38.088648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.088605 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" containerName="maas-api" Oct 31 15:18:38.088820 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.088740 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="4907085c-e7cb-4cec-a720-4fa41cf6060c" containerName="istio-proxy" Oct 31 15:18:38.088820 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.088752 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e55f7d9-6d54-485c-b8f5-f3b0a18c06cb" containerName="maas-api" Oct 31 15:18:38.092119 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.092097 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.153589 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.153545 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b62069-openshift-default-dockercfg-4lsj9\"" Oct 31 15:18:38.155082 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.154967 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f8748b-f6c8-4581-942e-e2a6524d6e97-istiod-ca-cert\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.155082 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.155034 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-certs\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.155722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.155459 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-socket\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.155722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.155529 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-podinfo\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.155722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.155572 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-token\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.155722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.155650 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-credential-socket\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.156648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.156103 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-data\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.156648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.156156 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-envoy\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.156648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.156213 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flkp6\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-kube-api-access-flkp6\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.158283 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.158260 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh"] Oct 31 15:18:38.257877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.257685 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-socket\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.257877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.257775 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-podinfo\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.257877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.257818 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-token\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.257883 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-credential-socket\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.257952 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-data\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.257995 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-envoy\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.258056 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flkp6\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-kube-api-access-flkp6\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.258088 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-socket\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.258158 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f8748b-f6c8-4581-942e-e2a6524d6e97-istiod-ca-cert\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.258214 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-certs\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.258254 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-credential-socket\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.258475 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-data\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.258896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.258679 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-certs\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.259000 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.258951 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f8748b-f6c8-4581-942e-e2a6524d6e97-istiod-ca-cert\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.260632 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.260609 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-podinfo\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.261051 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.261033 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-envoy\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.282122 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.282103 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:18:38.320538 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.320514 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-token\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.325383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.325350 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flkp6\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-kube-api-access-flkp6\") pod \"e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.358010 4399 generic.go:356] "Generic (PLEG): container finished" podID="b7cc0df2-9462-47b4-96f0-6107c0cdc28e" containerID="2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872" exitCode=0 Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.358190 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" event={"ID":"b7cc0df2-9462-47b4-96f0-6107c0cdc28e","Type":"ContainerDied","Data":"2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872"} Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.358239 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" event={"ID":"b7cc0df2-9462-47b4-96f0-6107c0cdc28e","Type":"ContainerDied","Data":"502dee6db943056a61f558c050b400820da34fa981e79c74586b4e804362ed9c"} Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.358271 4399 scope.go:122] "RemoveContainer" containerID="2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872" Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.358430 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw" Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.359391 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-metrics-certs\") pod \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.359459 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-kube-api-access-9cf25\" (UniqueName: \"kubernetes.io/projected/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-kube-api-access-9cf25\") pod \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.359647 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-openshift-service-ca\") pod \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.359741 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-maas-api-tls\") pod \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\" (UID: \"b7cc0df2-9462-47b4-96f0-6107c0cdc28e\") " Oct 31 15:18:38.366585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.361171 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-openshift-service-ca" pod "b7cc0df2-9462-47b4-96f0-6107c0cdc28e" (UID: "b7cc0df2-9462-47b4-96f0-6107c0cdc28e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:18:38.389725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.369247 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-metrics-certs" pod "b7cc0df2-9462-47b4-96f0-6107c0cdc28e" (UID: "b7cc0df2-9462-47b4-96f0-6107c0cdc28e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:18:38.389725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.375234 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-kube-api-access-9cf25" pod "b7cc0df2-9462-47b4-96f0-6107c0cdc28e" (UID: "b7cc0df2-9462-47b4-96f0-6107c0cdc28e"). InnerVolumeSpecName "kube-api-access-9cf25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:38.391391 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.390732 4399 scope.go:122] "RemoveContainer" containerID="2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872" Oct 31 15:18:38.391391 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:18:38.391264 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872\": container with ID starting with 2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872 not found: ID does not exist" containerID="2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872" Oct 31 15:18:38.391391 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.391299 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872"} err="failed to get container status \"2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872\": rpc error: code = NotFound desc = could not find container \"2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872\": container with ID starting with 2c3466d2f07e64cbd4f209bf59007c89f891f339402603303d1369e66aec6872 not found: ID does not exist" Oct 31 15:18:38.392762 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.392667 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-maas-api-tls" pod "b7cc0df2-9462-47b4-96f0-6107c0cdc28e" (UID: "b7cc0df2-9462-47b4-96f0-6107c0cdc28e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:18:38.412414 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.411970 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:38.465960 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.465921 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:38.466109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.465965 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:38.466109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.465983 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:38.466109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.465999 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cf25\" (UniqueName: \"kubernetes.io/projected/b7cc0df2-9462-47b4-96f0-6107c0cdc28e-kube-api-access-9cf25\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:38.492485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.492017 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df"] Oct 31 15:18:38.492485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.492266 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" podUID="a616bb05-16fb-40e1-a4f7-7c5a396662f5" containerName="payload-pre-processing" containerID="cri-o://220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b" gracePeriod=30 Oct 31 15:18:38.754490 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.754431 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:18:38.790169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.790132 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw"] Oct 31 15:18:38.794507 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.794476 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-644a8a-8488c7b7-8zcqw"] Oct 31 15:18:38.870592 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.870566 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a616bb05-16fb-40e1-a4f7-7c5a396662f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a616bb05-16fb-40e1-a4f7-7c5a396662f5-plugins-config-volume\") pod \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\" (UID: \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\") " Oct 31 15:18:38.870780 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.870614 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a616bb05-16fb-40e1-a4f7-7c5a396662f5-kube-api-access-4dr57\" (UniqueName: \"kubernetes.io/projected/a616bb05-16fb-40e1-a4f7-7c5a396662f5-kube-api-access-4dr57\") pod \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\" (UID: \"a616bb05-16fb-40e1-a4f7-7c5a396662f5\") " Oct 31 15:18:38.871007 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.870985 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a616bb05-16fb-40e1-a4f7-7c5a396662f5-plugins-config-volume" pod "a616bb05-16fb-40e1-a4f7-7c5a396662f5" (UID: "a616bb05-16fb-40e1-a4f7-7c5a396662f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:18:38.871260 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.871240 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a616bb05-16fb-40e1-a4f7-7c5a396662f5-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:38.872728 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.872692 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a616bb05-16fb-40e1-a4f7-7c5a396662f5-kube-api-access-4dr57" pod "a616bb05-16fb-40e1-a4f7-7c5a396662f5" (UID: "a616bb05-16fb-40e1-a4f7-7c5a396662f5"). InnerVolumeSpecName "kube-api-access-4dr57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:38.877845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.877818 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh"] Oct 31 15:18:38.879142 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:18:38.879124 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f8748b_f6c8_4581_942e_e2a6524d6e97.slice/crio-4e49a8aa32afd845f2e18a5241acb4034bbb0769bb9df65214f04259510d2463 WatchSource:0}: Error finding container 4e49a8aa32afd845f2e18a5241acb4034bbb0769bb9df65214f04259510d2463: Status 404 returned error can't find the container with id 4e49a8aa32afd845f2e18a5241acb4034bbb0769bb9df65214f04259510d2463 Oct 31 15:18:38.881607 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.881579 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:18:38.881691 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.881639 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:18:38.881691 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.881667 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:18:38.898557 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.898535 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7cc0df2-9462-47b4-96f0-6107c0cdc28e" path="/var/lib/kubelet/pods/b7cc0df2-9462-47b4-96f0-6107c0cdc28e/volumes" Oct 31 15:18:38.975209 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:38.975161 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dr57\" (UniqueName: \"kubernetes.io/projected/a616bb05-16fb-40e1-a4f7-7c5a396662f5-kube-api-access-4dr57\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:39.367218 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.367136 4399 generic.go:356] "Generic (PLEG): container finished" podID="a616bb05-16fb-40e1-a4f7-7c5a396662f5" containerID="220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b" exitCode=0 Oct 31 15:18:39.367218 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.367203 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" Oct 31 15:18:39.367428 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.367216 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" event={"ID":"a616bb05-16fb-40e1-a4f7-7c5a396662f5","Type":"ContainerDied","Data":"220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b"} Oct 31 15:18:39.367428 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.367273 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df" event={"ID":"a616bb05-16fb-40e1-a4f7-7c5a396662f5","Type":"ContainerDied","Data":"ad8a1d72fc28082ecf14da73d544e2bcadfc8e464af72b152c6fdcccbf8eeb90"} Oct 31 15:18:39.367428 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.367304 4399 scope.go:122] "RemoveContainer" containerID="220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b" Oct 31 15:18:39.370441 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.370414 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" event={"ID":"75f8748b-f6c8-4581-942e-e2a6524d6e97","Type":"ContainerStarted","Data":"9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf"} Oct 31 15:18:39.370587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.370459 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" event={"ID":"75f8748b-f6c8-4581-942e-e2a6524d6e97","Type":"ContainerStarted","Data":"4e49a8aa32afd845f2e18a5241acb4034bbb0769bb9df65214f04259510d2463"} Oct 31 15:18:39.377684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.377664 4399 scope.go:122] "RemoveContainer" containerID="220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b" Oct 31 15:18:39.378014 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:18:39.377993 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b\": container with ID starting with 220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b not found: ID does not exist" containerID="220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b" Oct 31 15:18:39.378117 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.378048 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b"} err="failed to get container status \"220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b\": rpc error: code = NotFound desc = could not find container \"220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b\": container with ID starting with 220d2e44a75f945c5057e20a177ccb933c934818fea2e50b7638f6b5ac35f21b not found: ID does not exist" Oct 31 15:18:39.413334 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.413306 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:39.530897 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.530732 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df"] Oct 31 15:18:39.533880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.533855 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-644a8a-f89485779-nb2df"] Oct 31 15:18:39.649709 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:39.649601 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" podStartSLOduration=1.6495884250000001 podStartE2EDuration="1.649588425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:18:39.645780932 +0000 UTC m=+1962.857022346" watchObservedRunningTime="2026-10-31 15:18:39.649588425 +0000 UTC m=+1962.860829836" Oct 31 15:18:40.418582 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:40.418553 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:40.897024 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:40.896996 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a616bb05-16fb-40e1-a4f7-7c5a396662f5" path="/var/lib/kubelet/pods/a616bb05-16fb-40e1-a4f7-7c5a396662f5/volumes" Oct 31 15:18:41.383420 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:41.383384 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:41.384540 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:41.384516 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:18:41.777192 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:41.777163 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp"] Oct 31 15:18:41.777647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:41.777396 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" containerID="cri-o://f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3" gracePeriod=30 Oct 31 15:18:46.664437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:46.664404 4399 patch_prober.go:29] interesting pod/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:18:46.664836 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:46.664456 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:18:47.087444 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.087410 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:18:47.195673 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.195642 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-token\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-token\") pod \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " Oct 31 15:18:47.195673 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.195676 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-socket\") pod \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " Oct 31 15:18:47.195950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.195780 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-kube-api-access-zb2gj\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-kube-api-access-zb2gj\") pod \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " Oct 31 15:18:47.195950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.195836 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-certs\") pod \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " Oct 31 15:18:47.195950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.195872 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-credential-socket\") pod \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " Oct 31 15:18:47.195950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.195918 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-podinfo\") pod \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " Oct 31 15:18:47.195950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.195946 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-data\") pod \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " Oct 31 15:18:47.196193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196022 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-envoy\") pod \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " Oct 31 15:18:47.196193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196069 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istiod-ca-cert\") pod \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\" (UID: \"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61\") " Oct 31 15:18:47.196193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196068 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-socket" pod "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" (UID: "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.196193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196124 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-certs" pod "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" (UID: "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.196193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196169 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-credential-socket" pod "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" (UID: "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.196455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196255 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-data" pod "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" (UID: "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.196877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196551 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istiod-ca-cert" pod "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" (UID: "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:18:47.196877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196778 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.196877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196800 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.196877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196816 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.196877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196829 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.196877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.196844 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.199854 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.199818 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-kube-api-access-zb2gj" pod "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" (UID: "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61"). InnerVolumeSpecName "kube-api-access-zb2gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:47.200299 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.200265 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-token" pod "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" (UID: "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:18:47.200649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.200591 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-envoy" pod "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" (UID: "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:18:47.200649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.200625 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-podinfo" pod "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" (UID: "4efc5957-2d8d-4a14-b4fb-8d6404ef3f61"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:18:47.297604 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.297524 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb2gj\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-kube-api-access-zb2gj\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.297604 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.297556 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.297604 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.297571 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.297604 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.297582 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:18:47.429674 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.429637 4399 generic.go:356] "Generic (PLEG): container finished" podID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerID="f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3" exitCode=0 Oct 31 15:18:47.429825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.429731 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" event={"ID":"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61","Type":"ContainerDied","Data":"f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3"} Oct 31 15:18:47.429825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.429741 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" Oct 31 15:18:47.429825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.429819 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp" event={"ID":"4efc5957-2d8d-4a14-b4fb-8d6404ef3f61","Type":"ContainerDied","Data":"5e40a74f256bdb3bd1324a2315827c4d52d1528aafbace15516aa6e8ade9c83d"} Oct 31 15:18:47.429943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.429845 4399 scope.go:122] "RemoveContainer" containerID="f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3" Oct 31 15:18:47.440437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.440420 4399 scope.go:122] "RemoveContainer" containerID="5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af" Oct 31 15:18:47.450290 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.450275 4399 scope.go:122] "RemoveContainer" containerID="f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3" Oct 31 15:18:47.450545 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:18:47.450523 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3\": container with ID starting with f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3 not found: ID does not exist" containerID="f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3" Oct 31 15:18:47.450613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.450553 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3"} err="failed to get container status \"f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3\": rpc error: code = NotFound desc = could not find container \"f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3\": container with ID starting with f4b5335615e06a5ba92d67b8b5b69161a481032ac932fd6f32836b53110a21b3 not found: ID does not exist" Oct 31 15:18:47.450613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.450570 4399 scope.go:122] "RemoveContainer" containerID="5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af" Oct 31 15:18:47.450899 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:18:47.450871 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af\": container with ID starting with 5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af not found: ID does not exist" containerID="5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af" Oct 31 15:18:47.450949 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.450909 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af"} err="failed to get container status \"5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af\": rpc error: code = NotFound desc = could not find container \"5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af\": container with ID starting with 5833601e8b222b403967210c3dad6c72096abc00dc246f4497f584ed3ddd23af not found: ID does not exist" Oct 31 15:18:47.594272 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.594182 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp"] Oct 31 15:18:47.598016 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:47.597991 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-644a8a-openshift-default-85b689b9b7-k9vtp"] Oct 31 15:18:48.897287 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:18:48.897257 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" path="/var/lib/kubelet/pods/4efc5957-2d8d-4a14-b4fb-8d6404ef3f61/volumes" Oct 31 15:18:56.210892 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:18:56.210845 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:19:01.046398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.046367 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5"] Oct 31 15:19:01.046977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.046963 4399 cpu_manager.go:414] "containerMap: removing container" podUID="a616bb05-16fb-40e1-a4f7-7c5a396662f5" containerName="payload-pre-processing" Oct 31 15:19:01.047038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.046978 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a616bb05-16fb-40e1-a4f7-7c5a396662f5" containerName="payload-pre-processing" Oct 31 15:19:01.047038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.046998 4399 cpu_manager.go:414] "containerMap: removing container" podUID="b7cc0df2-9462-47b4-96f0-6107c0cdc28e" containerName="maas-api" Oct 31 15:19:01.047038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.047004 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7cc0df2-9462-47b4-96f0-6107c0cdc28e" containerName="maas-api" Oct 31 15:19:01.047038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.047014 4399 cpu_manager.go:414] "containerMap: removing container" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" Oct 31 15:19:01.047038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.047019 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" Oct 31 15:19:01.047187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.047127 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a616bb05-16fb-40e1-a4f7-7c5a396662f5" containerName="payload-pre-processing" Oct 31 15:19:01.047187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.047136 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7cc0df2-9462-47b4-96f0-6107c0cdc28e" containerName="maas-api" Oct 31 15:19:01.047187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.047143 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" Oct 31 15:19:01.047187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.047153 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" Oct 31 15:19:01.052450 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.052434 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.107567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.107532 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5"] Oct 31 15:19:01.152418 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.152386 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-379d72-serving-cert\"" Oct 31 15:19:01.165058 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.165034 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-metrics-certs\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.165172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.165072 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9c559-fe10-4601-b79b-9d691fa990dc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.165172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.165095 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.165272 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.165252 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rsrn\" (UniqueName: \"kubernetes.io/projected/85f9c559-fe10-4601-b79b-9d691fa990dc-kube-api-access-6rsrn\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.267103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.267067 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-metrics-certs\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.267252 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.267117 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9c559-fe10-4601-b79b-9d691fa990dc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.267252 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.267148 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.267335 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:19:01.267271 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-serving-cert: secret "maas-api-e2e-praxis-a-379d72-serving-cert" not found Oct 31 15:19:01.267335 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.267306 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rsrn\" (UniqueName: \"kubernetes.io/projected/85f9c559-fe10-4601-b79b-9d691fa990dc-kube-api-access-6rsrn\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.267432 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:19:01.267354 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls podName:85f9c559-fe10-4601-b79b-9d691fa990dc nodeName:}" failed. No retries permitted until 2026-10-31 15:19:01.767334873 +0000 UTC m=+1984.978576270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls") pod "maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" (UID: "85f9c559-fe10-4601-b79b-9d691fa990dc") : secret "maas-api-e2e-praxis-a-379d72-serving-cert" not found Oct 31 15:19:01.267886 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.267861 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9c559-fe10-4601-b79b-9d691fa990dc-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.269671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.269650 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-metrics-certs\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.326376 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.326282 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rsrn\" (UniqueName: \"kubernetes.io/projected/85f9c559-fe10-4601-b79b-9d691fa990dc-kube-api-access-6rsrn\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.776103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.776062 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.778613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.778584 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls\") pod \"maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:01.966096 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:01.966069 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:02.344181 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:02.344152 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5"] Oct 31 15:19:02.346026 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:19:02.345996 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f9c559_fe10_4601_b79b_9d691fa990dc.slice/crio-1dc56d2cd0b7885cb6fc8a5e8d359c6f7583ddb6c231336a74607764fdf92eda WatchSource:0}: Error finding container 1dc56d2cd0b7885cb6fc8a5e8d359c6f7583ddb6c231336a74607764fdf92eda: Status 404 returned error can't find the container with id 1dc56d2cd0b7885cb6fc8a5e8d359c6f7583ddb6c231336a74607764fdf92eda Oct 31 15:19:02.531121 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:02.531084 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" event={"ID":"85f9c559-fe10-4601-b79b-9d691fa990dc","Type":"ContainerStarted","Data":"1dc56d2cd0b7885cb6fc8a5e8d359c6f7583ddb6c231336a74607764fdf92eda"} Oct 31 15:19:03.538822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:03.538786 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" event={"ID":"85f9c559-fe10-4601-b79b-9d691fa990dc","Type":"ContainerStarted","Data":"6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3"} Oct 31 15:19:03.539189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:03.538991 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:03.657041 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:03.656987 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" podStartSLOduration=1.827769618 podStartE2EDuration="2.656973472s" totalImagesPullingTime="829.203854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:19:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:19:03.653425991 +0000 UTC m=+1986.864667461" watchObservedRunningTime="2026-10-31 15:19:03.656973472 +0000 UTC m=+1986.868214917" Oct 31 15:19:09.551333 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:09.551307 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:19:26.208032 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:19:26.207994 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:19:47.188497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:47.188412 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv"] Oct 31 15:19:47.189014 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:47.188675 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" containerID="cri-o://7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08" gracePeriod=30 Oct 31 15:19:47.835650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:47.835619 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qllhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:19:47.835847 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:47.835679 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:19:48.596093 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:48.596055 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt"] Oct 31 15:19:48.596585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:48.596354 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" containerID="cri-o://b9011e43902fe2a6f03a528fa5013d64b9f46d4a9823a1d52b37bb6d23923a99" gracePeriod=30 Oct 31 15:19:52.448212 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.448186 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 15:19:52.512171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512111 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-token\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-token\") pod \"2818d837-fb1b-46f8-b50b-902ac273d9e6\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " Oct 31 15:19:52.512291 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512196 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-credential-socket\") pod \"2818d837-fb1b-46f8-b50b-902ac273d9e6\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " Oct 31 15:19:52.512291 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512244 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-podinfo\") pod \"2818d837-fb1b-46f8-b50b-902ac273d9e6\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " Oct 31 15:19:52.512385 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512303 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2818d837-fb1b-46f8-b50b-902ac273d9e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2818d837-fb1b-46f8-b50b-902ac273d9e6-istiod-ca-cert\") pod \"2818d837-fb1b-46f8-b50b-902ac273d9e6\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " Oct 31 15:19:52.512423 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512386 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-data\") pod \"2818d837-fb1b-46f8-b50b-902ac273d9e6\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " Oct 31 15:19:52.512423 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512414 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-kube-api-access-m2gms\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-kube-api-access-m2gms\") pod \"2818d837-fb1b-46f8-b50b-902ac273d9e6\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " Oct 31 15:19:52.512527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512439 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-envoy\") pod \"2818d837-fb1b-46f8-b50b-902ac273d9e6\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " Oct 31 15:19:52.512527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512467 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-socket\") pod \"2818d837-fb1b-46f8-b50b-902ac273d9e6\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " Oct 31 15:19:52.512527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512500 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-certs\") pod \"2818d837-fb1b-46f8-b50b-902ac273d9e6\" (UID: \"2818d837-fb1b-46f8-b50b-902ac273d9e6\") " Oct 31 15:19:52.512527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512510 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-credential-socket" pod "2818d837-fb1b-46f8-b50b-902ac273d9e6" (UID: "2818d837-fb1b-46f8-b50b-902ac273d9e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.512896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512783 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-data" pod "2818d837-fb1b-46f8-b50b-902ac273d9e6" (UID: "2818d837-fb1b-46f8-b50b-902ac273d9e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.512896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512863 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-socket" pod "2818d837-fb1b-46f8-b50b-902ac273d9e6" (UID: "2818d837-fb1b-46f8-b50b-902ac273d9e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.512896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.512806 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2818d837-fb1b-46f8-b50b-902ac273d9e6-istiod-ca-cert" pod "2818d837-fb1b-46f8-b50b-902ac273d9e6" (UID: "2818d837-fb1b-46f8-b50b-902ac273d9e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:52.513124 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.513092 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.513124 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.513108 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2818d837-fb1b-46f8-b50b-902ac273d9e6-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.513124 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.513117 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.513124 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.513125 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.513449 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.513211 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-certs" pod "2818d837-fb1b-46f8-b50b-902ac273d9e6" (UID: "2818d837-fb1b-46f8-b50b-902ac273d9e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.514658 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.514634 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-token" pod "2818d837-fb1b-46f8-b50b-902ac273d9e6" (UID: "2818d837-fb1b-46f8-b50b-902ac273d9e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:52.515226 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.515198 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-kube-api-access-m2gms" pod "2818d837-fb1b-46f8-b50b-902ac273d9e6" (UID: "2818d837-fb1b-46f8-b50b-902ac273d9e6"). InnerVolumeSpecName "kube-api-access-m2gms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:52.515475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.515455 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-envoy" pod "2818d837-fb1b-46f8-b50b-902ac273d9e6" (UID: "2818d837-fb1b-46f8-b50b-902ac273d9e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:52.515544 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.515494 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-podinfo" pod "2818d837-fb1b-46f8-b50b-902ac273d9e6" (UID: "2818d837-fb1b-46f8-b50b-902ac273d9e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:19:52.614629 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.614594 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2gms\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-kube-api-access-m2gms\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.614629 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.614626 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.614842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.614641 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2818d837-fb1b-46f8-b50b-902ac273d9e6-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.614842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.614653 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.614842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.614665 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2818d837-fb1b-46f8-b50b-902ac273d9e6-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:52.861032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.860946 4399 generic.go:356] "Generic (PLEG): container finished" podID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerID="7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08" exitCode=0 Oct 31 15:19:52.861032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.861002 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" Oct 31 15:19:52.861245 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.861029 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" event={"ID":"2818d837-fb1b-46f8-b50b-902ac273d9e6","Type":"ContainerDied","Data":"7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08"} Oct 31 15:19:52.861245 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.861083 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv" event={"ID":"2818d837-fb1b-46f8-b50b-902ac273d9e6","Type":"ContainerDied","Data":"851dc19a65deccf4c66cde2028a85dbe33c0ed6e4770bf2b91096e7a3c45dcb0"} Oct 31 15:19:52.861245 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.861105 4399 scope.go:122] "RemoveContainer" containerID="7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08" Oct 31 15:19:52.872050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.872031 4399 scope.go:122] "RemoveContainer" containerID="7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08" Oct 31 15:19:52.872346 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:19:52.872322 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08\": container with ID starting with 7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08 not found: ID does not exist" containerID="7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08" Oct 31 15:19:52.872403 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:52.872356 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08"} err="failed to get container status \"7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08\": rpc error: code = NotFound desc = could not find container \"7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08\": container with ID starting with 7bd1d4f949a49a6a1f21dcebb50025250917db11965fd40756b578cb67130c08 not found: ID does not exist" Oct 31 15:19:53.025568 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.025523 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv"] Oct 31 15:19:53.033849 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.031005 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qllhv"] Oct 31 15:19:53.868892 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.868820 4399 generic.go:356] "Generic (PLEG): container finished" podID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerID="b9011e43902fe2a6f03a528fa5013d64b9f46d4a9823a1d52b37bb6d23923a99" exitCode=0 Oct 31 15:19:53.869249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.868900 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" event={"ID":"ccdd1426-2ca4-436b-89a4-924805c3f544","Type":"ContainerDied","Data":"b9011e43902fe2a6f03a528fa5013d64b9f46d4a9823a1d52b37bb6d23923a99"} Oct 31 15:19:53.869249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.868980 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" event={"ID":"ccdd1426-2ca4-436b-89a4-924805c3f544","Type":"ContainerDied","Data":"ef633800c87c8a6904425f76410ac29f5fc4828f9dc87e5d9043f8b9b3882745"} Oct 31 15:19:53.869249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.868998 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef633800c87c8a6904425f76410ac29f5fc4828f9dc87e5d9043f8b9b3882745" Oct 31 15:19:53.873897 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.873878 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:19:53.930996 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.930973 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-data\") pod \"ccdd1426-2ca4-436b-89a4-924805c3f544\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " Oct 31 15:19:53.931126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931010 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-envoy\") pod \"ccdd1426-2ca4-436b-89a4-924805c3f544\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " Oct 31 15:19:53.931126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931040 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-credential-socket\") pod \"ccdd1426-2ca4-436b-89a4-924805c3f544\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " Oct 31 15:19:53.931213 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931149 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-token\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-token\") pod \"ccdd1426-2ca4-436b-89a4-924805c3f544\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " Oct 31 15:19:53.931213 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931168 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-data" pod "ccdd1426-2ca4-436b-89a4-924805c3f544" (UID: "ccdd1426-2ca4-436b-89a4-924805c3f544"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:53.931291 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931211 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-socket\") pod \"ccdd1426-2ca4-436b-89a4-924805c3f544\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " Oct 31 15:19:53.931330 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931289 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccdd1426-2ca4-436b-89a4-924805c3f544-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdd1426-2ca4-436b-89a4-924805c3f544-istiod-ca-cert\") pod \"ccdd1426-2ca4-436b-89a4-924805c3f544\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " Oct 31 15:19:53.931374 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931357 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-podinfo\") pod \"ccdd1426-2ca4-436b-89a4-924805c3f544\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " Oct 31 15:19:53.931430 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931409 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-kube-api-access-j4xbq\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-kube-api-access-j4xbq\") pod \"ccdd1426-2ca4-436b-89a4-924805c3f544\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " Oct 31 15:19:53.931430 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931413 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-credential-socket" pod "ccdd1426-2ca4-436b-89a4-924805c3f544" (UID: "ccdd1426-2ca4-436b-89a4-924805c3f544"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:53.931579 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931446 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-certs\") pod \"ccdd1426-2ca4-436b-89a4-924805c3f544\" (UID: \"ccdd1426-2ca4-436b-89a4-924805c3f544\") " Oct 31 15:19:53.931579 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931563 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-socket" pod "ccdd1426-2ca4-436b-89a4-924805c3f544" (UID: "ccdd1426-2ca4-436b-89a4-924805c3f544"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:53.932159 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.931998 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:53.932159 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.932029 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:53.932159 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.932031 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccdd1426-2ca4-436b-89a4-924805c3f544-istiod-ca-cert" pod "ccdd1426-2ca4-436b-89a4-924805c3f544" (UID: "ccdd1426-2ca4-436b-89a4-924805c3f544"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:53.932159 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.932044 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:53.932159 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.932062 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-certs" pod "ccdd1426-2ca4-436b-89a4-924805c3f544" (UID: "ccdd1426-2ca4-436b-89a4-924805c3f544"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:53.933281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.933249 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-envoy" pod "ccdd1426-2ca4-436b-89a4-924805c3f544" (UID: "ccdd1426-2ca4-436b-89a4-924805c3f544"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:19:53.933575 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.933558 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-token" pod "ccdd1426-2ca4-436b-89a4-924805c3f544" (UID: "ccdd1426-2ca4-436b-89a4-924805c3f544"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:53.933661 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.933626 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-kube-api-access-j4xbq" pod "ccdd1426-2ca4-436b-89a4-924805c3f544" (UID: "ccdd1426-2ca4-436b-89a4-924805c3f544"). InnerVolumeSpecName "kube-api-access-j4xbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:53.933765 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:53.933750 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-podinfo" pod "ccdd1426-2ca4-436b-89a4-924805c3f544" (UID: "ccdd1426-2ca4-436b-89a4-924805c3f544"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:19:54.033358 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:54.033322 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:54.033504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:54.033365 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:54.033504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:54.033379 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccdd1426-2ca4-436b-89a4-924805c3f544-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:54.033504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:54.033393 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccdd1426-2ca4-436b-89a4-924805c3f544-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:54.033504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:54.033485 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4xbq\" (UniqueName: \"kubernetes.io/projected/ccdd1426-2ca4-436b-89a4-924805c3f544-kube-api-access-j4xbq\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:54.033653 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:54.033503 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccdd1426-2ca4-436b-89a4-924805c3f544-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:54.874838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:54.874807 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt" Oct 31 15:19:54.900525 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:54.900494 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" path="/var/lib/kubelet/pods/2818d837-fb1b-46f8-b50b-902ac273d9e6/volumes" Oct 31 15:19:55.090398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:55.090366 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt"] Oct 31 15:19:55.139672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:55.139597 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lnpkt"] Oct 31 15:19:56.209790 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:19:56.209747 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:19:56.896171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:56.896141 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" path="/var/lib/kubelet/pods/ccdd1426-2ca4-436b-89a4-924805c3f544/volumes" Oct 31 15:19:57.129248 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.129212 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn"] Oct 31 15:19:57.129473 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.129437 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" podUID="8da85bae-56c8-429c-a5df-e48336d6bb7e" containerName="maas-api" containerID="cri-o://a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724" gracePeriod=30 Oct 31 15:19:57.411663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.411638 4399 scope.go:122] "RemoveContainer" containerID="b9011e43902fe2a6f03a528fa5013d64b9f46d4a9823a1d52b37bb6d23923a99" Oct 31 15:19:57.427476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.427455 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:19:57.580300 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.580266 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da85bae-56c8-429c-a5df-e48336d6bb7e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da85bae-56c8-429c-a5df-e48336d6bb7e-openshift-service-ca\") pod \"8da85bae-56c8-429c-a5df-e48336d6bb7e\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " Oct 31 15:19:57.580509 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.580383 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-metrics-certs\") pod \"8da85bae-56c8-429c-a5df-e48336d6bb7e\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " Oct 31 15:19:57.580509 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.580461 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-maas-api-tls\") pod \"8da85bae-56c8-429c-a5df-e48336d6bb7e\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " Oct 31 15:19:57.581012 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.580734 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da85bae-56c8-429c-a5df-e48336d6bb7e-openshift-service-ca" pod "8da85bae-56c8-429c-a5df-e48336d6bb7e" (UID: "8da85bae-56c8-429c-a5df-e48336d6bb7e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:57.581012 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.580823 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da85bae-56c8-429c-a5df-e48336d6bb7e-kube-api-access-2xg8b\" (UniqueName: \"kubernetes.io/projected/8da85bae-56c8-429c-a5df-e48336d6bb7e-kube-api-access-2xg8b\") pod \"8da85bae-56c8-429c-a5df-e48336d6bb7e\" (UID: \"8da85bae-56c8-429c-a5df-e48336d6bb7e\") " Oct 31 15:19:57.581402 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.581373 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da85bae-56c8-429c-a5df-e48336d6bb7e-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.587761 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.585864 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-metrics-certs" pod "8da85bae-56c8-429c-a5df-e48336d6bb7e" (UID: "8da85bae-56c8-429c-a5df-e48336d6bb7e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:19:57.587761 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.585891 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-maas-api-tls" pod "8da85bae-56c8-429c-a5df-e48336d6bb7e" (UID: "8da85bae-56c8-429c-a5df-e48336d6bb7e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:19:57.587761 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.585930 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da85bae-56c8-429c-a5df-e48336d6bb7e-kube-api-access-2xg8b" pod "8da85bae-56c8-429c-a5df-e48336d6bb7e" (UID: "8da85bae-56c8-429c-a5df-e48336d6bb7e"). InnerVolumeSpecName "kube-api-access-2xg8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:57.683366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.683276 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xg8b\" (UniqueName: \"kubernetes.io/projected/8da85bae-56c8-429c-a5df-e48336d6bb7e-kube-api-access-2xg8b\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.683366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.683308 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.683366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.683318 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da85bae-56c8-429c-a5df-e48336d6bb7e-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:57.895479 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.895443 4399 generic.go:356] "Generic (PLEG): container finished" podID="8da85bae-56c8-429c-a5df-e48336d6bb7e" containerID="a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724" exitCode=0 Oct 31 15:19:57.895641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.895485 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" Oct 31 15:19:57.895641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.895521 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" event={"ID":"8da85bae-56c8-429c-a5df-e48336d6bb7e","Type":"ContainerDied","Data":"a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724"} Oct 31 15:19:57.895641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.895565 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn" event={"ID":"8da85bae-56c8-429c-a5df-e48336d6bb7e","Type":"ContainerDied","Data":"d8332745835440ad002bce9a57a54dae262825cb72271c6edd7568cc75d8d945"} Oct 31 15:19:57.895641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.895639 4399 scope.go:122] "RemoveContainer" containerID="a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724" Oct 31 15:19:57.907397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.907379 4399 scope.go:122] "RemoveContainer" containerID="a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724" Oct 31 15:19:57.907654 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:19:57.907637 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724\": container with ID starting with a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724 not found: ID does not exist" containerID="a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724" Oct 31 15:19:57.907715 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:57.907662 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724"} err="failed to get container status \"a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724\": rpc error: code = NotFound desc = could not find container \"a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724\": container with ID starting with a370d183f09f4836f83f2a833b417d47c8b5546e8e35124af74c3ea1627b1724 not found: ID does not exist" Oct 31 15:19:58.065833 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.065797 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn"] Oct 31 15:19:58.071918 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.071843 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-94a889-5798548757-xxxcn"] Oct 31 15:19:58.312948 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.312917 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v"] Oct 31 15:19:58.313197 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.313173 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" podUID="16c062f9-11cb-494d-ac86-bd88848eb94e" containerName="maas-api" containerID="cri-o://4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718" gracePeriod=30 Oct 31 15:19:58.609045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.609024 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:19:58.699135 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.699102 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls\") pod \"16c062f9-11cb-494d-ac86-bd88848eb94e\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " Oct 31 15:19:58.699281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.699148 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16c062f9-11cb-494d-ac86-bd88848eb94e-kube-api-access-jv7pw\" (UniqueName: \"kubernetes.io/projected/16c062f9-11cb-494d-ac86-bd88848eb94e-kube-api-access-jv7pw\") pod \"16c062f9-11cb-494d-ac86-bd88848eb94e\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " Oct 31 15:19:58.699281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.699207 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16c062f9-11cb-494d-ac86-bd88848eb94e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c062f9-11cb-494d-ac86-bd88848eb94e-openshift-service-ca\") pod \"16c062f9-11cb-494d-ac86-bd88848eb94e\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " Oct 31 15:19:58.699425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.699342 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-metrics-certs\") pod \"16c062f9-11cb-494d-ac86-bd88848eb94e\" (UID: \"16c062f9-11cb-494d-ac86-bd88848eb94e\") " Oct 31 15:19:58.699607 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.699582 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16c062f9-11cb-494d-ac86-bd88848eb94e-openshift-service-ca" pod "16c062f9-11cb-494d-ac86-bd88848eb94e" (UID: "16c062f9-11cb-494d-ac86-bd88848eb94e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:19:58.699832 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.699814 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/16c062f9-11cb-494d-ac86-bd88848eb94e-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:58.701430 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.701397 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls" pod "16c062f9-11cb-494d-ac86-bd88848eb94e" (UID: "16c062f9-11cb-494d-ac86-bd88848eb94e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:19:58.701529 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.701437 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16c062f9-11cb-494d-ac86-bd88848eb94e-kube-api-access-jv7pw" pod "16c062f9-11cb-494d-ac86-bd88848eb94e" (UID: "16c062f9-11cb-494d-ac86-bd88848eb94e"). InnerVolumeSpecName "kube-api-access-jv7pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:19:58.701529 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.701513 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-metrics-certs" pod "16c062f9-11cb-494d-ac86-bd88848eb94e" (UID: "16c062f9-11cb-494d-ac86-bd88848eb94e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:19:58.800897 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.800864 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:58.800897 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.800900 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/16c062f9-11cb-494d-ac86-bd88848eb94e-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:58.801104 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.800916 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv7pw\" (UniqueName: \"kubernetes.io/projected/16c062f9-11cb-494d-ac86-bd88848eb94e-kube-api-access-jv7pw\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:19:58.901564 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.901446 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da85bae-56c8-429c-a5df-e48336d6bb7e" path="/var/lib/kubelet/pods/8da85bae-56c8-429c-a5df-e48336d6bb7e/volumes" Oct 31 15:19:58.907156 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.907126 4399 generic.go:356] "Generic (PLEG): container finished" podID="16c062f9-11cb-494d-ac86-bd88848eb94e" containerID="4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718" exitCode=0 Oct 31 15:19:58.907429 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.907189 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" Oct 31 15:19:58.907616 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.907183 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" event={"ID":"16c062f9-11cb-494d-ac86-bd88848eb94e","Type":"ContainerDied","Data":"4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718"} Oct 31 15:19:58.907616 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.907534 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v" event={"ID":"16c062f9-11cb-494d-ac86-bd88848eb94e","Type":"ContainerDied","Data":"9a26a8ac3d14a63c7d1b5663386d8bf800cd9639dc2005951746bc827d461994"} Oct 31 15:19:58.907616 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.907575 4399 scope.go:122] "RemoveContainer" containerID="4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718" Oct 31 15:19:58.921258 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.921240 4399 scope.go:122] "RemoveContainer" containerID="4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718" Oct 31 15:19:58.921538 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:19:58.921516 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718\": container with ID starting with 4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718 not found: ID does not exist" containerID="4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718" Oct 31 15:19:58.921629 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:58.921544 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718"} err="failed to get container status \"4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718\": rpc error: code = NotFound desc = could not find container \"4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718\": container with ID starting with 4a1f5fb631a08fd9c791e6e4dade3b14ed2b5a76bc6747905c94a869ac2cb718 not found: ID does not exist" Oct 31 15:19:59.071058 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:59.071032 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v"] Oct 31 15:19:59.074045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:19:59.074020 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f76cce-8548978f9b-97q6v"] Oct 31 15:20:00.577130 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:00.577097 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8"] Oct 31 15:20:00.577579 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:00.577331 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" containerID="cri-o://4a3b781a6e3980a36849982da36642d7c01d8d41cf72333f0723229757780542" gracePeriod=30 Oct 31 15:20:00.897509 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:00.897435 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16c062f9-11cb-494d-ac86-bd88848eb94e" path="/var/lib/kubelet/pods/16c062f9-11cb-494d-ac86-bd88848eb94e/volumes" Oct 31 15:20:01.590196 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:01.590160 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:20:01.590599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:01.590212 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:20:05.952438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:05.952404 4399 generic.go:356] "Generic (PLEG): container finished" podID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerID="4a3b781a6e3980a36849982da36642d7c01d8d41cf72333f0723229757780542" exitCode=0 Oct 31 15:20:05.952834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:05.952473 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" event={"ID":"7dffa02c-662a-4798-b5c8-e5b5d08ef462","Type":"ContainerDied","Data":"4a3b781a6e3980a36849982da36642d7c01d8d41cf72333f0723229757780542"} Oct 31 15:20:06.251799 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.251768 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:20:06.399072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399044 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-credential-socket\") pod \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " Oct 31 15:20:06.399072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399078 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-kube-api-access-6kj6h\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-kube-api-access-6kj6h\") pod \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " Oct 31 15:20:06.399309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399121 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-envoy\") pod \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " Oct 31 15:20:06.399309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399151 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-socket\") pod \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " Oct 31 15:20:06.399309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399212 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-certs\") pod \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " Oct 31 15:20:06.399309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399296 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-data\") pod \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " Oct 31 15:20:06.399518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399347 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istiod-ca-cert\") pod \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " Oct 31 15:20:06.399518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399349 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-credential-socket" pod "7dffa02c-662a-4798-b5c8-e5b5d08ef462" (UID: "7dffa02c-662a-4798-b5c8-e5b5d08ef462"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:06.399518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399396 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-podinfo\") pod \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " Oct 31 15:20:06.399518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399446 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-token\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-token\") pod \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\" (UID: \"7dffa02c-662a-4798-b5c8-e5b5d08ef462\") " Oct 31 15:20:06.399518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399503 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-socket" pod "7dffa02c-662a-4798-b5c8-e5b5d08ef462" (UID: "7dffa02c-662a-4798-b5c8-e5b5d08ef462"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:06.399824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399598 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-certs" pod "7dffa02c-662a-4798-b5c8-e5b5d08ef462" (UID: "7dffa02c-662a-4798-b5c8-e5b5d08ef462"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:06.399824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399652 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-data" pod "7dffa02c-662a-4798-b5c8-e5b5d08ef462" (UID: "7dffa02c-662a-4798-b5c8-e5b5d08ef462"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:06.399824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.399791 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istiod-ca-cert" pod "7dffa02c-662a-4798-b5c8-e5b5d08ef462" (UID: "7dffa02c-662a-4798-b5c8-e5b5d08ef462"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:06.400053 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.400029 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:06.400173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.400056 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:06.400173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.400071 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:06.400173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.400083 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:06.400173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.400095 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:06.401842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.401822 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-token" pod "7dffa02c-662a-4798-b5c8-e5b5d08ef462" (UID: "7dffa02c-662a-4798-b5c8-e5b5d08ef462"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:06.402086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.402070 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-podinfo" pod "7dffa02c-662a-4798-b5c8-e5b5d08ef462" (UID: "7dffa02c-662a-4798-b5c8-e5b5d08ef462"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:20:06.402163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.402142 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-kube-api-access-6kj6h" pod "7dffa02c-662a-4798-b5c8-e5b5d08ef462" (UID: "7dffa02c-662a-4798-b5c8-e5b5d08ef462"). InnerVolumeSpecName "kube-api-access-6kj6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:06.402426 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.402399 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-envoy" pod "7dffa02c-662a-4798-b5c8-e5b5d08ef462" (UID: "7dffa02c-662a-4798-b5c8-e5b5d08ef462"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:06.501424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.501304 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:06.501424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.501334 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:06.501424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.501343 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kj6h\" (UniqueName: \"kubernetes.io/projected/7dffa02c-662a-4798-b5c8-e5b5d08ef462-kube-api-access-6kj6h\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:06.501424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.501352 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dffa02c-662a-4798-b5c8-e5b5d08ef462-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:06.959931 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.959905 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" Oct 31 15:20:06.960361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.959899 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8" event={"ID":"7dffa02c-662a-4798-b5c8-e5b5d08ef462","Type":"ContainerDied","Data":"8689160224debf24036bae6022f428b02e6e5b174e4682bb8e4499ac2ed06fb8"} Oct 31 15:20:06.960361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:06.960040 4399 scope.go:122] "RemoveContainer" containerID="4a3b781a6e3980a36849982da36642d7c01d8d41cf72333f0723229757780542" Oct 31 15:20:07.127311 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:07.127283 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8"] Oct 31 15:20:07.175962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:07.175882 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-8h7c8"] Oct 31 15:20:08.896393 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:08.896350 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" path="/var/lib/kubelet/pods/7dffa02c-662a-4798-b5c8-e5b5d08ef462/volumes" Oct 31 15:20:12.046846 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:12.046812 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z"] Oct 31 15:20:17.051103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:17.051023 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" podUID="a2ee0e99-7f27-4287-8253-3cb74a89f626" containerName="payload-pre-processing" containerID="cri-o://5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6" gracePeriod=25 Oct 31 15:20:17.368470 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:17.368441 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:20:17.449159 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:17.449125 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2ee0e99-7f27-4287-8253-3cb74a89f626-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee0e99-7f27-4287-8253-3cb74a89f626-plugins-config-volume\") pod \"a2ee0e99-7f27-4287-8253-3cb74a89f626\" (UID: \"a2ee0e99-7f27-4287-8253-3cb74a89f626\") " Oct 31 15:20:17.449350 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:17.449293 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2ee0e99-7f27-4287-8253-3cb74a89f626-kube-api-access-7bjz9\" (UniqueName: \"kubernetes.io/projected/a2ee0e99-7f27-4287-8253-3cb74a89f626-kube-api-access-7bjz9\") pod \"a2ee0e99-7f27-4287-8253-3cb74a89f626\" (UID: \"a2ee0e99-7f27-4287-8253-3cb74a89f626\") " Oct 31 15:20:17.449585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:17.449558 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2ee0e99-7f27-4287-8253-3cb74a89f626-plugins-config-volume" pod "a2ee0e99-7f27-4287-8253-3cb74a89f626" (UID: "a2ee0e99-7f27-4287-8253-3cb74a89f626"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:17.450130 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:17.450105 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2ee0e99-7f27-4287-8253-3cb74a89f626-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:17.452022 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:17.451995 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2ee0e99-7f27-4287-8253-3cb74a89f626-kube-api-access-7bjz9" pod "a2ee0e99-7f27-4287-8253-3cb74a89f626" (UID: "a2ee0e99-7f27-4287-8253-3cb74a89f626"). InnerVolumeSpecName "kube-api-access-7bjz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:17.552822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:17.552768 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bjz9\" (UniqueName: \"kubernetes.io/projected/a2ee0e99-7f27-4287-8253-3cb74a89f626-kube-api-access-7bjz9\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:18.088734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.078350 4399 generic.go:356] "Generic (PLEG): container finished" podID="a2ee0e99-7f27-4287-8253-3cb74a89f626" containerID="5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6" exitCode=0 Oct 31 15:20:18.092934 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.078460 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" Oct 31 15:20:18.093158 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.092765 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" event={"ID":"a2ee0e99-7f27-4287-8253-3cb74a89f626","Type":"ContainerDied","Data":"5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6"} Oct 31 15:20:18.093248 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.093220 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z" event={"ID":"a2ee0e99-7f27-4287-8253-3cb74a89f626","Type":"ContainerDied","Data":"4be5acfc383d1bbd2c7060d1b423ddefa14be4767f938703143b43c0e43b6dac"} Oct 31 15:20:18.093315 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.093274 4399 scope.go:122] "RemoveContainer" containerID="5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6" Oct 31 15:20:18.129810 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.129788 4399 scope.go:122] "RemoveContainer" containerID="5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6" Oct 31 15:20:18.137950 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:20:18.137763 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6\": container with ID starting with 5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6 not found: ID does not exist" containerID="5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6" Oct 31 15:20:18.137950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.137808 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6"} err="failed to get container status \"5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6\": rpc error: code = NotFound desc = could not find container \"5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6\": container with ID starting with 5e557963b3dc69b16e532775a2e00020c95d1d04416e70763b1c1d68e0c273d6 not found: ID does not exist" Oct 31 15:20:18.266546 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.266495 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z"] Oct 31 15:20:18.311001 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.310970 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9cc557-786859b99-smf9z"] Oct 31 15:20:18.896070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:18.896035 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2ee0e99-7f27-4287-8253-3cb74a89f626" path="/var/lib/kubelet/pods/a2ee0e99-7f27-4287-8253-3cb74a89f626/volumes" Oct 31 15:20:26.210725 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:20:26.210584 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:20:26.579052 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:26.577797 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": read tcp 10.133.0.2:35812->10.133.0.109:15021: read: connection reset by peer" start-of-body= Oct 31 15:20:26.579052 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:26.577870 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": read tcp 10.133.0.2:35812->10.133.0.109:15021: read: connection reset by peer" Oct 31 15:20:27.159360 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:27.159329 4399 generic.go:356] "Generic (PLEG): container finished" podID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerID="9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf" exitCode=0 Oct 31 15:20:27.159535 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:27.159388 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" event={"ID":"75f8748b-f6c8-4581-942e-e2a6524d6e97","Type":"ContainerDied","Data":"9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf"} Oct 31 15:20:27.159768 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:27.159753 4399 scope.go:122] "RemoveContainer" containerID="9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf" Oct 31 15:20:27.161247 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:27.161219 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:20:27.161307 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:27.161288 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:20:27.161343 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:27.161322 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:20:28.171425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:28.171386 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" event={"ID":"75f8748b-f6c8-4581-942e-e2a6524d6e97","Type":"ContainerStarted","Data":"bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6"} Oct 31 15:20:28.412518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:28.412482 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:20:28.412860 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:28.412781 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 15:20:28.412860 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:28.412825 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 15:20:29.413581 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:29.413536 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 15:20:29.414002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:29.413598 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 15:20:30.412780 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:30.412749 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 15:20:30.412964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:30.412810 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 15:20:31.413157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:31.413114 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 15:20:31.413588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:31.413169 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 15:20:31.606676 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:31.606631 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl"] Oct 31 15:20:31.606943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:31.606914 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" podUID="ac2df8a3-04d0-407f-a4e6-269f49073e0f" containerName="maas-api" containerID="cri-o://854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47" gracePeriod=30 Oct 31 15:20:31.930762 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:31.930738 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:20:32.004518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.004438 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac2df8a3-04d0-407f-a4e6-269f49073e0f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2df8a3-04d0-407f-a4e6-269f49073e0f-openshift-service-ca\") pod \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " Oct 31 15:20:32.004518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.004493 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac2df8a3-04d0-407f-a4e6-269f49073e0f-kube-api-access-scvdw\" (UniqueName: \"kubernetes.io/projected/ac2df8a3-04d0-407f-a4e6-269f49073e0f-kube-api-access-scvdw\") pod \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " Oct 31 15:20:32.004783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.004559 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls\") pod \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " Oct 31 15:20:32.004783 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.004747 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-metrics-certs\") pod \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\" (UID: \"ac2df8a3-04d0-407f-a4e6-269f49073e0f\") " Oct 31 15:20:32.004894 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.004836 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2df8a3-04d0-407f-a4e6-269f49073e0f-openshift-service-ca" pod "ac2df8a3-04d0-407f-a4e6-269f49073e0f" (UID: "ac2df8a3-04d0-407f-a4e6-269f49073e0f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:32.005337 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.005315 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac2df8a3-04d0-407f-a4e6-269f49073e0f-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:32.007187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.007163 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2df8a3-04d0-407f-a4e6-269f49073e0f-kube-api-access-scvdw" pod "ac2df8a3-04d0-407f-a4e6-269f49073e0f" (UID: "ac2df8a3-04d0-407f-a4e6-269f49073e0f"). InnerVolumeSpecName "kube-api-access-scvdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:32.008375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.008347 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls" pod "ac2df8a3-04d0-407f-a4e6-269f49073e0f" (UID: "ac2df8a3-04d0-407f-a4e6-269f49073e0f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:20:32.008599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.008576 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-metrics-certs" pod "ac2df8a3-04d0-407f-a4e6-269f49073e0f" (UID: "ac2df8a3-04d0-407f-a4e6-269f49073e0f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:20:32.107348 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.107316 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scvdw\" (UniqueName: \"kubernetes.io/projected/ac2df8a3-04d0-407f-a4e6-269f49073e0f-kube-api-access-scvdw\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:32.107348 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.107349 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:32.107348 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.107360 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac2df8a3-04d0-407f-a4e6-269f49073e0f-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:32.197748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.197708 4399 generic.go:356] "Generic (PLEG): container finished" podID="ac2df8a3-04d0-407f-a4e6-269f49073e0f" containerID="854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47" exitCode=0 Oct 31 15:20:32.197928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.197793 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" Oct 31 15:20:32.197928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.197823 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" event={"ID":"ac2df8a3-04d0-407f-a4e6-269f49073e0f","Type":"ContainerDied","Data":"854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47"} Oct 31 15:20:32.197928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.197893 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl" event={"ID":"ac2df8a3-04d0-407f-a4e6-269f49073e0f","Type":"ContainerDied","Data":"bc48faf0f8f92cb502e78962e13e0b5c93550c58ce2231966099fc3dd1762461"} Oct 31 15:20:32.197928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.197915 4399 scope.go:122] "RemoveContainer" containerID="854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47" Oct 31 15:20:32.212163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.210393 4399 scope.go:122] "RemoveContainer" containerID="854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47" Oct 31 15:20:32.212163 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:20:32.211985 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47\": container with ID starting with 854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47 not found: ID does not exist" containerID="854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47" Oct 31 15:20:32.212163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.212013 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47"} err="failed to get container status \"854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47\": rpc error: code = NotFound desc = could not find container \"854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47\": container with ID starting with 854cb9ddce0a4dc8c2c9ce57a6b3bd40ee55bec05630dd95ce18a9829e7bdf47 not found: ID does not exist" Oct 31 15:20:32.367785 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.367676 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl"] Oct 31 15:20:32.371443 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.371407 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-032b05-7c779ff88c-949kl"] Oct 31 15:20:32.413556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.413522 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 31 15:20:32.413912 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.413577 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 31 15:20:32.896058 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:32.896021 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac2df8a3-04d0-407f-a4e6-269f49073e0f" path="/var/lib/kubelet/pods/ac2df8a3-04d0-407f-a4e6-269f49073e0f/volumes" Oct 31 15:20:33.458502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:33.458457 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:20:33.459286 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:33.459255 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:20:33.460129 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:33.460105 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:20:38.409104 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:38.409068 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft"] Oct 31 15:20:38.409545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:38.409268 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" containerID="cri-o://ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c" gracePeriod=30 Oct 31 15:20:43.673680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.673654 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:20:43.743112 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743086 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea262732-121b-484c-bcc7-cfe929b2147f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea262732-121b-484c-bcc7-cfe929b2147f-istio-podinfo\") pod \"ea262732-121b-484c-bcc7-cfe929b2147f\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " Oct 31 15:20:43.743241 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743170 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-data\") pod \"ea262732-121b-484c-bcc7-cfe929b2147f\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " Oct 31 15:20:43.743241 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743209 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-certs\") pod \"ea262732-121b-484c-bcc7-cfe929b2147f\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " Oct 31 15:20:43.743352 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743247 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-socket\") pod \"ea262732-121b-484c-bcc7-cfe929b2147f\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " Oct 31 15:20:43.743409 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743395 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-kube-api-access-nnqgk\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-kube-api-access-nnqgk\") pod \"ea262732-121b-484c-bcc7-cfe929b2147f\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " Oct 31 15:20:43.743463 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743433 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-credential-socket\") pod \"ea262732-121b-484c-bcc7-cfe929b2147f\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " Oct 31 15:20:43.743520 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743465 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-envoy\") pod \"ea262732-121b-484c-bcc7-cfe929b2147f\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " Oct 31 15:20:43.743520 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743450 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-data" pod "ea262732-121b-484c-bcc7-cfe929b2147f" (UID: "ea262732-121b-484c-bcc7-cfe929b2147f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:43.743520 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743505 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea262732-121b-484c-bcc7-cfe929b2147f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea262732-121b-484c-bcc7-cfe929b2147f-istiod-ca-cert\") pod \"ea262732-121b-484c-bcc7-cfe929b2147f\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " Oct 31 15:20:43.743670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743524 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-certs" pod "ea262732-121b-484c-bcc7-cfe929b2147f" (UID: "ea262732-121b-484c-bcc7-cfe929b2147f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:43.743670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743554 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-socket" pod "ea262732-121b-484c-bcc7-cfe929b2147f" (UID: "ea262732-121b-484c-bcc7-cfe929b2147f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:43.743670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743559 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-istio-token\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-istio-token\") pod \"ea262732-121b-484c-bcc7-cfe929b2147f\" (UID: \"ea262732-121b-484c-bcc7-cfe929b2147f\") " Oct 31 15:20:43.743877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743724 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-credential-socket" pod "ea262732-121b-484c-bcc7-cfe929b2147f" (UID: "ea262732-121b-484c-bcc7-cfe929b2147f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:43.743962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.743940 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea262732-121b-484c-bcc7-cfe929b2147f-istiod-ca-cert" pod "ea262732-121b-484c-bcc7-cfe929b2147f" (UID: "ea262732-121b-484c-bcc7-cfe929b2147f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:20:43.744192 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.744167 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:43.744266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.744192 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea262732-121b-484c-bcc7-cfe929b2147f-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:43.744266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.744207 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:43.744266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.744220 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:43.744266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.744233 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:43.745738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.745681 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-envoy" pod "ea262732-121b-484c-bcc7-cfe929b2147f" (UID: "ea262732-121b-484c-bcc7-cfe929b2147f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:20:43.745827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.745758 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-kube-api-access-nnqgk" pod "ea262732-121b-484c-bcc7-cfe929b2147f" (UID: "ea262732-121b-484c-bcc7-cfe929b2147f"). InnerVolumeSpecName "kube-api-access-nnqgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:43.746044 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.746017 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea262732-121b-484c-bcc7-cfe929b2147f-istio-podinfo" pod "ea262732-121b-484c-bcc7-cfe929b2147f" (UID: "ea262732-121b-484c-bcc7-cfe929b2147f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:20:43.746098 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.746054 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-istio-token" pod "ea262732-121b-484c-bcc7-cfe929b2147f" (UID: "ea262732-121b-484c-bcc7-cfe929b2147f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:20:43.845753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.845723 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnqgk\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-kube-api-access-nnqgk\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:43.845753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.845753 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea262732-121b-484c-bcc7-cfe929b2147f-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:43.845968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.845769 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea262732-121b-484c-bcc7-cfe929b2147f-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:43.845968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:43.845782 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea262732-121b-484c-bcc7-cfe929b2147f-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:20:44.274106 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.274055 4399 generic.go:356] "Generic (PLEG): container finished" podID="ea262732-121b-484c-bcc7-cfe929b2147f" containerID="ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c" exitCode=0 Oct 31 15:20:44.274106 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.274098 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" event={"ID":"ea262732-121b-484c-bcc7-cfe929b2147f","Type":"ContainerDied","Data":"ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c"} Oct 31 15:20:44.274369 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.274144 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" Oct 31 15:20:44.274369 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.274161 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft" event={"ID":"ea262732-121b-484c-bcc7-cfe929b2147f","Type":"ContainerDied","Data":"d0dd306d9e4768b82fe3e9e2a53d75ac4e3def81415e29bf89d7457ee61477fa"} Oct 31 15:20:44.274369 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.274202 4399 scope.go:122] "RemoveContainer" containerID="ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c" Oct 31 15:20:44.285907 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.285887 4399 scope.go:122] "RemoveContainer" containerID="25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51" Oct 31 15:20:44.295257 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.295240 4399 scope.go:122] "RemoveContainer" containerID="ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c" Oct 31 15:20:44.295547 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:20:44.295528 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c\": container with ID starting with ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c not found: ID does not exist" containerID="ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c" Oct 31 15:20:44.295595 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.295556 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c"} err="failed to get container status \"ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c\": rpc error: code = NotFound desc = could not find container \"ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c\": container with ID starting with ed6a76aededf00194dc0dbfbbf4f84d394d95f568f09c5bb62708bfe004b186c not found: ID does not exist" Oct 31 15:20:44.295595 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.295575 4399 scope.go:122] "RemoveContainer" containerID="25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51" Oct 31 15:20:44.295835 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:20:44.295817 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51\": container with ID starting with 25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51 not found: ID does not exist" containerID="25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51" Oct 31 15:20:44.295885 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.295842 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51"} err="failed to get container status \"25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51\": rpc error: code = NotFound desc = could not find container \"25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51\": container with ID starting with 25959bfcd90193c3a09ffe4a69d17c3b8a0b35c9f88741362d836e7d4689ce51 not found: ID does not exist" Oct 31 15:20:44.436980 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.436947 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft"] Oct 31 15:20:44.486800 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.486764 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-032b05-openshift-default-7db7b54655-7mwft"] Oct 31 15:20:44.896560 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:20:44.896529 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" path="/var/lib/kubelet/pods/ea262732-121b-484c-bcc7-cfe929b2147f/volumes" Oct 31 15:20:56.206963 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:20:56.206929 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:21:09.751116 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751036 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4"] Oct 31 15:21:09.751628 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751613 4399 cpu_manager.go:414] "containerMap: removing container" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" Oct 31 15:21:09.751628 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751629 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efc5957-2d8d-4a14-b4fb-8d6404ef3f61" containerName="istio-proxy" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751648 4399 cpu_manager.go:414] "containerMap: removing container" podUID="16c062f9-11cb-494d-ac86-bd88848eb94e" containerName="maas-api" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751653 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16c062f9-11cb-494d-ac86-bd88848eb94e" containerName="maas-api" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751667 4399 cpu_manager.go:414] "containerMap: removing container" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751671 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751679 4399 cpu_manager.go:414] "containerMap: removing container" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751684 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751707 4399 cpu_manager.go:414] "containerMap: removing container" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751712 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751720 4399 cpu_manager.go:414] "containerMap: removing container" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751724 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" Oct 31 15:21:09.751731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751734 4399 cpu_manager.go:414] "containerMap: removing container" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751738 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751749 4399 cpu_manager.go:414] "containerMap: removing container" podUID="ac2df8a3-04d0-407f-a4e6-269f49073e0f" containerName="maas-api" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751754 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac2df8a3-04d0-407f-a4e6-269f49073e0f" containerName="maas-api" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751769 4399 cpu_manager.go:414] "containerMap: removing container" podUID="a2ee0e99-7f27-4287-8253-3cb74a89f626" containerName="payload-pre-processing" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751773 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2ee0e99-7f27-4287-8253-3cb74a89f626" containerName="payload-pre-processing" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751791 4399 cpu_manager.go:414] "containerMap: removing container" podUID="8da85bae-56c8-429c-a5df-e48336d6bb7e" containerName="maas-api" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751795 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da85bae-56c8-429c-a5df-e48336d6bb7e" containerName="maas-api" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751876 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2ee0e99-7f27-4287-8253-3cb74a89f626" containerName="payload-pre-processing" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751886 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da85bae-56c8-429c-a5df-e48336d6bb7e" containerName="maas-api" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751893 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="16c062f9-11cb-494d-ac86-bd88848eb94e" containerName="maas-api" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751899 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dffa02c-662a-4798-b5c8-e5b5d08ef462" containerName="istio-proxy" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751906 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751912 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccdd1426-2ca4-436b-89a4-924805c3f544" containerName="istio-proxy" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751918 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="2818d837-fb1b-46f8-b50b-902ac273d9e6" containerName="istio-proxy" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751923 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea262732-121b-484c-bcc7-cfe929b2147f" containerName="istio-proxy" Oct 31 15:21:09.752108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.751928 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac2df8a3-04d0-407f-a4e6-269f49073e0f" containerName="maas-api" Oct 31 15:21:09.757015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.756998 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.809085 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.809057 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-d9cddd-serving-cert\"" Oct 31 15:21:09.845241 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.845207 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l954f\" (UniqueName: \"kubernetes.io/projected/d1c38736-f7c8-47b2-bb7d-3cff37743d02-kube-api-access-l954f\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.845396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.845375 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.845498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.845481 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-metrics-certs\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.845544 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.845511 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c38736-f7c8-47b2-bb7d-3cff37743d02-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.861214 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.861187 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4"] Oct 31 15:21:09.947091 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.947059 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-metrics-certs\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.947274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.947102 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c38736-f7c8-47b2-bb7d-3cff37743d02-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.947274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.947168 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l954f\" (UniqueName: \"kubernetes.io/projected/d1c38736-f7c8-47b2-bb7d-3cff37743d02-kube-api-access-l954f\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.947274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.947239 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.947460 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:09.947378 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-serving-cert: secret "maas-api-e2e-praxis-a-d9cddd-serving-cert" not found Oct 31 15:21:09.947460 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:09.947447 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls podName:d1c38736-f7c8-47b2-bb7d-3cff37743d02 nodeName:}" failed. No retries permitted until 2026-10-31 15:21:10.447429308 +0000 UTC m=+2113.658670698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls") pod "maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" (UID: "d1c38736-f7c8-47b2-bb7d-3cff37743d02") : secret "maas-api-e2e-praxis-a-d9cddd-serving-cert" not found Oct 31 15:21:09.948122 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.948098 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c38736-f7c8-47b2-bb7d-3cff37743d02-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:09.950118 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:09.950093 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-metrics-certs\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:10.006982 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:10.006922 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l954f\" (UniqueName: \"kubernetes.io/projected/d1c38736-f7c8-47b2-bb7d-3cff37743d02-kube-api-access-l954f\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:10.454893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:10.454863 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:10.457375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:10.457355 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls\") pod \"maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:10.670550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:10.670517 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:11.047715 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:11.047666 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4"] Oct 31 15:21:11.048593 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:21:11.048564 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c38736_f7c8_47b2_bb7d_3cff37743d02.slice/crio-fda5e04792f898e350bf165fcffd0a00502b2e8f7df6cb7f5416a4c6ffb1ec6e WatchSource:0}: Error finding container fda5e04792f898e350bf165fcffd0a00502b2e8f7df6cb7f5416a4c6ffb1ec6e: Status 404 returned error can't find the container with id fda5e04792f898e350bf165fcffd0a00502b2e8f7df6cb7f5416a4c6ffb1ec6e Oct 31 15:21:11.430871 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:11.430791 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" event={"ID":"d1c38736-f7c8-47b2-bb7d-3cff37743d02","Type":"ContainerStarted","Data":"fda5e04792f898e350bf165fcffd0a00502b2e8f7df6cb7f5416a4c6ffb1ec6e"} Oct 31 15:21:12.438556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:12.438516 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" event={"ID":"d1c38736-f7c8-47b2-bb7d-3cff37743d02","Type":"ContainerStarted","Data":"9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806"} Oct 31 15:21:12.439057 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:12.438687 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:12.552602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:12.552545 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" podStartSLOduration=2.6742315679999997 podStartE2EDuration="3.552531427s" totalImagesPullingTime="878.299859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:21:12.550292707 +0000 UTC m=+2115.761534120" watchObservedRunningTime="2026-10-31 15:21:12.552531427 +0000 UTC m=+2115.763772838" Oct 31 15:21:12.831907 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:12.831870 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr"] Oct 31 15:21:15.033387 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.033355 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj"] Oct 31 15:21:15.037503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.037486 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.091088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.091060 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-15cc44-serving-cert\"" Oct 31 15:21:15.143506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.143467 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj"] Oct 31 15:21:15.215067 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.215027 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-metrics-certs\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.215428 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.215242 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.215428 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.215321 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhgq\" (UniqueName: \"kubernetes.io/projected/8f67d7b4-0b14-4390-bc24-58ce77918f57-kube-api-access-wmhgq\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.215428 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.215366 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f67d7b4-0b14-4390-bc24-58ce77918f57-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.316354 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.316256 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.316354 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.316319 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhgq\" (UniqueName: \"kubernetes.io/projected/8f67d7b4-0b14-4390-bc24-58ce77918f57-kube-api-access-wmhgq\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.316597 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.316390 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f67d7b4-0b14-4390-bc24-58ce77918f57-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.316597 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:15.316407 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-serving-cert: secret "maas-api-e2e-praxis-b-15cc44-serving-cert" not found Oct 31 15:21:15.316597 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.316448 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-metrics-certs\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.316597 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:15.316478 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls podName:8f67d7b4-0b14-4390-bc24-58ce77918f57 nodeName:}" failed. No retries permitted until 2026-10-31 15:21:15.816461406 +0000 UTC m=+2119.027702797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls") pod "maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" (UID: "8f67d7b4-0b14-4390-bc24-58ce77918f57") : secret "maas-api-e2e-praxis-b-15cc44-serving-cert" not found Oct 31 15:21:15.317086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.317065 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f67d7b4-0b14-4390-bc24-58ce77918f57-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.319079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.319056 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-metrics-certs\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.374240 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.374210 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhgq\" (UniqueName: \"kubernetes.io/projected/8f67d7b4-0b14-4390-bc24-58ce77918f57-kube-api-access-wmhgq\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.822664 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.822627 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.825530 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.825501 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls\") pod \"maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:15.950635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:15.950606 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:16.327134 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:16.327111 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj"] Oct 31 15:21:16.329806 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:21:16.329777 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f67d7b4_0b14_4390_bc24_58ce77918f57.slice/crio-841430fbddbf471fba26943dba15317e3eae404d91a25df5bb67e0cc2190eca2 WatchSource:0}: Error finding container 841430fbddbf471fba26943dba15317e3eae404d91a25df5bb67e0cc2190eca2: Status 404 returned error can't find the container with id 841430fbddbf471fba26943dba15317e3eae404d91a25df5bb67e0cc2190eca2 Oct 31 15:21:16.465762 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:16.465728 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" event={"ID":"8f67d7b4-0b14-4390-bc24-58ce77918f57","Type":"ContainerStarted","Data":"841430fbddbf471fba26943dba15317e3eae404d91a25df5bb67e0cc2190eca2"} Oct 31 15:21:17.474211 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:17.474173 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" event={"ID":"8f67d7b4-0b14-4390-bc24-58ce77918f57","Type":"ContainerStarted","Data":"95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c"} Oct 31 15:21:17.474689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:17.474258 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:17.635418 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:17.635362 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" podStartSLOduration=2.828827104 podStartE2EDuration="3.635348944s" totalImagesPullingTime="806.52184ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:21:17.633128651 +0000 UTC m=+2120.844370065" watchObservedRunningTime="2026-10-31 15:21:17.635348944 +0000 UTC m=+2120.846590357" Oct 31 15:21:17.832547 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:17.832460 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" podUID="e448a9d5-75a8-488a-bb87-3108abf4ac15" containerName="payload-processing" containerID="cri-o://b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463" gracePeriod=25 Oct 31 15:21:18.108265 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.108231 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:21:18.251748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.251682 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e448a9d5-75a8-488a-bb87-3108abf4ac15-kube-api-access-lv6m2\" (UniqueName: \"kubernetes.io/projected/e448a9d5-75a8-488a-bb87-3108abf4ac15-kube-api-access-lv6m2\") pod \"e448a9d5-75a8-488a-bb87-3108abf4ac15\" (UID: \"e448a9d5-75a8-488a-bb87-3108abf4ac15\") " Oct 31 15:21:18.251945 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.251758 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e448a9d5-75a8-488a-bb87-3108abf4ac15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e448a9d5-75a8-488a-bb87-3108abf4ac15-plugins-config-volume\") pod \"e448a9d5-75a8-488a-bb87-3108abf4ac15\" (UID: \"e448a9d5-75a8-488a-bb87-3108abf4ac15\") " Oct 31 15:21:18.252481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.252455 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e448a9d5-75a8-488a-bb87-3108abf4ac15-plugins-config-volume" pod "e448a9d5-75a8-488a-bb87-3108abf4ac15" (UID: "e448a9d5-75a8-488a-bb87-3108abf4ac15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:21:18.254953 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.254926 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e448a9d5-75a8-488a-bb87-3108abf4ac15-kube-api-access-lv6m2" pod "e448a9d5-75a8-488a-bb87-3108abf4ac15" (UID: "e448a9d5-75a8-488a-bb87-3108abf4ac15"). InnerVolumeSpecName "kube-api-access-lv6m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:21:18.353599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.353532 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv6m2\" (UniqueName: \"kubernetes.io/projected/e448a9d5-75a8-488a-bb87-3108abf4ac15-kube-api-access-lv6m2\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:18.353599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.353562 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e448a9d5-75a8-488a-bb87-3108abf4ac15-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:18.450065 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.450035 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:21:18.481756 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.481725 4399 generic.go:356] "Generic (PLEG): container finished" podID="e448a9d5-75a8-488a-bb87-3108abf4ac15" containerID="b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463" exitCode=0 Oct 31 15:21:18.482207 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.481768 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" Oct 31 15:21:18.482207 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.481806 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" event={"ID":"e448a9d5-75a8-488a-bb87-3108abf4ac15","Type":"ContainerDied","Data":"b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463"} Oct 31 15:21:18.482207 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.481857 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr" event={"ID":"e448a9d5-75a8-488a-bb87-3108abf4ac15","Type":"ContainerDied","Data":"175e6e15470a9955a072e8c621be08d7da2917cc3d046259f2353d912a531011"} Oct 31 15:21:18.482207 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.481881 4399 scope.go:122] "RemoveContainer" containerID="b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463" Oct 31 15:21:18.492118 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.492102 4399 scope.go:122] "RemoveContainer" containerID="b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463" Oct 31 15:21:18.492373 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:18.492354 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463\": container with ID starting with b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463 not found: ID does not exist" containerID="b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463" Oct 31 15:21:18.492443 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.492387 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463"} err="failed to get container status \"b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463\": rpc error: code = NotFound desc = could not find container \"b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463\": container with ID starting with b4d66296941574047785fbdd3f1db1d698449ec8fa019ae8365b30efa2a42463 not found: ID does not exist" Oct 31 15:21:18.674078 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.674008 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr"] Oct 31 15:21:18.681304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.681276 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-9ab5df-b7b95c9f7-525dr"] Oct 31 15:21:18.897108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:18.897077 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e448a9d5-75a8-488a-bb87-3108abf4ac15" path="/var/lib/kubelet/pods/e448a9d5-75a8-488a-bb87-3108abf4ac15/volumes" Oct 31 15:21:23.488206 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:23.488174 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:21:26.206662 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:26.206628 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:21:29.362711 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.362657 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn"] Oct 31 15:21:29.363263 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.363248 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e448a9d5-75a8-488a-bb87-3108abf4ac15" containerName="payload-processing" Oct 31 15:21:29.363309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.363265 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e448a9d5-75a8-488a-bb87-3108abf4ac15" containerName="payload-processing" Oct 31 15:21:29.363394 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.363385 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e448a9d5-75a8-488a-bb87-3108abf4ac15" containerName="payload-processing" Oct 31 15:21:29.366904 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.366888 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.420534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.420447 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-da9920-serving-cert\"" Oct 31 15:21:29.424104 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.424077 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn"] Oct 31 15:21:29.481614 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.481575 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-metrics-certs\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.482075 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.481754 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.482220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.482186 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ccd829-3732-4e8a-a8c5-883e788bede6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.482293 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.482248 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5wzp\" (UniqueName: \"kubernetes.io/projected/e6ccd829-3732-4e8a-a8c5-883e788bede6-kube-api-access-l5wzp\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.583253 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.583213 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5wzp\" (UniqueName: \"kubernetes.io/projected/e6ccd829-3732-4e8a-a8c5-883e788bede6-kube-api-access-l5wzp\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.583464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.583318 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-metrics-certs\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.583464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.583369 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.583464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.583434 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ccd829-3732-4e8a-a8c5-883e788bede6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.583643 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:29.583540 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-serving-cert: secret "maas-api-e2e-ipp-a-da9920-serving-cert" not found Oct 31 15:21:29.583643 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:29.583611 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls podName:e6ccd829-3732-4e8a-a8c5-883e788bede6 nodeName:}" failed. No retries permitted until 2026-10-31 15:21:30.083596006 +0000 UTC m=+2133.294837399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls") pod "maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" (UID: "e6ccd829-3732-4e8a-a8c5-883e788bede6") : secret "maas-api-e2e-ipp-a-da9920-serving-cert" not found Oct 31 15:21:29.584210 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.584189 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ccd829-3732-4e8a-a8c5-883e788bede6-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.586186 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.586155 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-metrics-certs\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:29.640715 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:29.640629 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5wzp\" (UniqueName: \"kubernetes.io/projected/e6ccd829-3732-4e8a-a8c5-883e788bede6-kube-api-access-l5wzp\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:30.089747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:30.089690 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:30.092517 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:30.092493 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls\") pod \"maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:30.281190 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:30.281161 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:30.658531 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:30.655798 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn"] Oct 31 15:21:30.661893 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:21:30.661867 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ccd829_3732_4e8a_a8c5_883e788bede6.slice/crio-78d44004beab28c9653b7a4830b536f4aba53fbfadf78ffb53294315760b8b40 WatchSource:0}: Error finding container 78d44004beab28c9653b7a4830b536f4aba53fbfadf78ffb53294315760b8b40: Status 404 returned error can't find the container with id 78d44004beab28c9653b7a4830b536f4aba53fbfadf78ffb53294315760b8b40 Oct 31 15:21:31.559305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:31.559268 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" event={"ID":"e6ccd829-3732-4e8a-a8c5-883e788bede6","Type":"ContainerStarted","Data":"78d44004beab28c9653b7a4830b536f4aba53fbfadf78ffb53294315760b8b40"} Oct 31 15:21:32.568566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:32.568529 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" event={"ID":"e6ccd829-3732-4e8a-a8c5-883e788bede6","Type":"ContainerStarted","Data":"25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14"} Oct 31 15:21:32.569082 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:32.568591 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:32.683385 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:32.683334 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" podStartSLOduration=2.750598251 podStartE2EDuration="3.683320126s" totalImagesPullingTime="932.721875ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:21:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:21:32.681234697 +0000 UTC m=+2135.892476110" watchObservedRunningTime="2026-10-31 15:21:32.683320126 +0000 UTC m=+2135.894561537" Oct 31 15:21:38.512899 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.512865 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f"] Oct 31 15:21:38.513355 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.513094 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" podUID="0d406da3-3311-47a8-b443-679c059505a6" containerName="maas-api" containerID="cri-o://4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2" gracePeriod=30 Oct 31 15:21:38.583661 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.583635 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:21:38.795496 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.795472 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:21:38.886482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.886443 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls\") pod \"0d406da3-3311-47a8-b443-679c059505a6\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " Oct 31 15:21:38.887147 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.887109 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d406da3-3311-47a8-b443-679c059505a6-kube-api-access-57s7z\" (UniqueName: \"kubernetes.io/projected/0d406da3-3311-47a8-b443-679c059505a6-kube-api-access-57s7z\") pod \"0d406da3-3311-47a8-b443-679c059505a6\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " Oct 31 15:21:38.887364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.887345 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d406da3-3311-47a8-b443-679c059505a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d406da3-3311-47a8-b443-679c059505a6-openshift-service-ca\") pod \"0d406da3-3311-47a8-b443-679c059505a6\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " Oct 31 15:21:38.888008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.887884 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d406da3-3311-47a8-b443-679c059505a6-openshift-service-ca" pod "0d406da3-3311-47a8-b443-679c059505a6" (UID: "0d406da3-3311-47a8-b443-679c059505a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:21:38.888284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.888265 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-metrics-certs\") pod \"0d406da3-3311-47a8-b443-679c059505a6\" (UID: \"0d406da3-3311-47a8-b443-679c059505a6\") " Oct 31 15:21:38.892221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.891969 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d406da3-3311-47a8-b443-679c059505a6-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:38.892221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.892169 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d406da3-3311-47a8-b443-679c059505a6-kube-api-access-57s7z" pod "0d406da3-3311-47a8-b443-679c059505a6" (UID: "0d406da3-3311-47a8-b443-679c059505a6"). InnerVolumeSpecName "kube-api-access-57s7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:21:38.893753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.893680 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls" pod "0d406da3-3311-47a8-b443-679c059505a6" (UID: "0d406da3-3311-47a8-b443-679c059505a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:21:38.894215 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.894175 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-metrics-certs" pod "0d406da3-3311-47a8-b443-679c059505a6" (UID: "0d406da3-3311-47a8-b443-679c059505a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:21:38.993118 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.993084 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:38.993118 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.993109 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0d406da3-3311-47a8-b443-679c059505a6-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:38.993118 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:38.993118 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57s7z\" (UniqueName: \"kubernetes.io/projected/0d406da3-3311-47a8-b443-679c059505a6-kube-api-access-57s7z\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:39.614443 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.614408 4399 generic.go:356] "Generic (PLEG): container finished" podID="0d406da3-3311-47a8-b443-679c059505a6" containerID="4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2" exitCode=0 Oct 31 15:21:39.614932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.614452 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" Oct 31 15:21:39.614932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.614492 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" event={"ID":"0d406da3-3311-47a8-b443-679c059505a6","Type":"ContainerDied","Data":"4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2"} Oct 31 15:21:39.614932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.614539 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f" event={"ID":"0d406da3-3311-47a8-b443-679c059505a6","Type":"ContainerDied","Data":"09ce6ed2e5e448cf185f96f50035c33a544c6483ca7baf070bb13552e148d474"} Oct 31 15:21:39.614932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.614560 4399 scope.go:122] "RemoveContainer" containerID="4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2" Oct 31 15:21:39.623662 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.623645 4399 scope.go:122] "RemoveContainer" containerID="4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2" Oct 31 15:21:39.623929 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:39.623910 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2\": container with ID starting with 4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2 not found: ID does not exist" containerID="4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2" Oct 31 15:21:39.623987 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.623942 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2"} err="failed to get container status \"4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2\": rpc error: code = NotFound desc = could not find container \"4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2\": container with ID starting with 4abb9502874113ff1455bcc65ee8333dd3ede00674d2c3d65c6222115ec4a4b2 not found: ID does not exist" Oct 31 15:21:39.658218 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.658195 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm"] Oct 31 15:21:39.658374 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.658357 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" podUID="ae8969a7-0130-4522-8955-549f2e16ae15" containerName="maas-api" containerID="cri-o://36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18" gracePeriod=30 Oct 31 15:21:39.778174 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.778141 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f"] Oct 31 15:21:39.780832 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.780805 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-9ab5df-579f4d59dd-7nb5f"] Oct 31 15:21:39.951842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:39.951809 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:21:40.002851 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.002822 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8969a7-0130-4522-8955-549f2e16ae15-kube-api-access-5bv4m\" (UniqueName: \"kubernetes.io/projected/ae8969a7-0130-4522-8955-549f2e16ae15-kube-api-access-5bv4m\") pod \"ae8969a7-0130-4522-8955-549f2e16ae15\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " Oct 31 15:21:40.003037 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.002905 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls\") pod \"ae8969a7-0130-4522-8955-549f2e16ae15\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " Oct 31 15:21:40.003087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.003032 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8969a7-0130-4522-8955-549f2e16ae15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8969a7-0130-4522-8955-549f2e16ae15-openshift-service-ca\") pod \"ae8969a7-0130-4522-8955-549f2e16ae15\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " Oct 31 15:21:40.003139 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.003115 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-metrics-certs\") pod \"ae8969a7-0130-4522-8955-549f2e16ae15\" (UID: \"ae8969a7-0130-4522-8955-549f2e16ae15\") " Oct 31 15:21:40.003499 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.003466 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8969a7-0130-4522-8955-549f2e16ae15-openshift-service-ca" pod "ae8969a7-0130-4522-8955-549f2e16ae15" (UID: "ae8969a7-0130-4522-8955-549f2e16ae15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:21:40.003856 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.003598 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae8969a7-0130-4522-8955-549f2e16ae15-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:40.005263 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.005233 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls" pod "ae8969a7-0130-4522-8955-549f2e16ae15" (UID: "ae8969a7-0130-4522-8955-549f2e16ae15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:21:40.005785 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.005756 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-metrics-certs" pod "ae8969a7-0130-4522-8955-549f2e16ae15" (UID: "ae8969a7-0130-4522-8955-549f2e16ae15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:21:40.005890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.005797 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8969a7-0130-4522-8955-549f2e16ae15-kube-api-access-5bv4m" pod "ae8969a7-0130-4522-8955-549f2e16ae15" (UID: "ae8969a7-0130-4522-8955-549f2e16ae15"). InnerVolumeSpecName "kube-api-access-5bv4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:21:40.104680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.104646 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:40.104680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.104682 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae8969a7-0130-4522-8955-549f2e16ae15-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:40.104909 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.104721 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bv4m\" (UniqueName: \"kubernetes.io/projected/ae8969a7-0130-4522-8955-549f2e16ae15-kube-api-access-5bv4m\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:21:40.632266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.631776 4399 generic.go:356] "Generic (PLEG): container finished" podID="ae8969a7-0130-4522-8955-549f2e16ae15" containerID="36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18" exitCode=0 Oct 31 15:21:40.632266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.631823 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" event={"ID":"ae8969a7-0130-4522-8955-549f2e16ae15","Type":"ContainerDied","Data":"36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18"} Oct 31 15:21:40.632266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.631875 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" event={"ID":"ae8969a7-0130-4522-8955-549f2e16ae15","Type":"ContainerDied","Data":"ee258d3666caf26792ac65650ed0657126e2b6a1225682c1366dac4f44292014"} Oct 31 15:21:40.632266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.631904 4399 scope.go:122] "RemoveContainer" containerID="36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18" Oct 31 15:21:40.632266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.632012 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm" Oct 31 15:21:40.663865 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.663837 4399 scope.go:122] "RemoveContainer" containerID="36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18" Oct 31 15:21:40.664480 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:40.664449 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18\": container with ID starting with 36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18 not found: ID does not exist" containerID="36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18" Oct 31 15:21:40.664582 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.664493 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18"} err="failed to get container status \"36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18\": rpc error: code = NotFound desc = could not find container \"36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18\": container with ID starting with 36e32022bd0d4e76d2d990651f2e7646d2ca8043725a51878359036d994e5d18 not found: ID does not exist" Oct 31 15:21:40.809917 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.809378 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm"] Oct 31 15:21:40.817381 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.817342 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ad5bfe-7dd8888d57-4v7gm"] Oct 31 15:21:40.896097 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.896028 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d406da3-3311-47a8-b443-679c059505a6" path="/var/lib/kubelet/pods/0d406da3-3311-47a8-b443-679c059505a6/volumes" Oct 31 15:21:40.896480 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:21:40.896466 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8969a7-0130-4522-8955-549f2e16ae15" path="/var/lib/kubelet/pods/ae8969a7-0130-4522-8955-549f2e16ae15/volumes" Oct 31 15:21:56.207679 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:21:56.207646 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:22:26.207264 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:22:26.207229 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:22:53.504080 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:53.504044 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh"] Oct 31 15:22:53.504498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:53.504250 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" containerID="cri-o://bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6" gracePeriod=30 Oct 31 15:22:54.303416 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.303373 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5"] Oct 31 15:22:54.303618 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.303585 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" podUID="85f9c559-fe10-4601-b79b-9d691fa990dc" containerName="maas-api" containerID="cri-o://6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3" gracePeriod=30 Oct 31 15:22:54.575578 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.575547 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:22:54.695485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.695454 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls\") pod \"85f9c559-fe10-4601-b79b-9d691fa990dc\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " Oct 31 15:22:54.695691 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.695524 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f9c559-fe10-4601-b79b-9d691fa990dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9c559-fe10-4601-b79b-9d691fa990dc-openshift-service-ca\") pod \"85f9c559-fe10-4601-b79b-9d691fa990dc\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " Oct 31 15:22:54.695691 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.695600 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-metrics-certs\") pod \"85f9c559-fe10-4601-b79b-9d691fa990dc\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " Oct 31 15:22:54.695838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.695692 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f9c559-fe10-4601-b79b-9d691fa990dc-kube-api-access-6rsrn\" (UniqueName: \"kubernetes.io/projected/85f9c559-fe10-4601-b79b-9d691fa990dc-kube-api-access-6rsrn\") pod \"85f9c559-fe10-4601-b79b-9d691fa990dc\" (UID: \"85f9c559-fe10-4601-b79b-9d691fa990dc\") " Oct 31 15:22:54.696311 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.696253 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f9c559-fe10-4601-b79b-9d691fa990dc-openshift-service-ca" pod "85f9c559-fe10-4601-b79b-9d691fa990dc" (UID: "85f9c559-fe10-4601-b79b-9d691fa990dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:22:54.701567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.701535 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-metrics-certs" pod "85f9c559-fe10-4601-b79b-9d691fa990dc" (UID: "85f9c559-fe10-4601-b79b-9d691fa990dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:22:54.706741 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.703476 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f9c559-fe10-4601-b79b-9d691fa990dc-kube-api-access-6rsrn" pod "85f9c559-fe10-4601-b79b-9d691fa990dc" (UID: "85f9c559-fe10-4601-b79b-9d691fa990dc"). InnerVolumeSpecName "kube-api-access-6rsrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:22:54.706741 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.703526 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls" pod "85f9c559-fe10-4601-b79b-9d691fa990dc" (UID: "85f9c559-fe10-4601-b79b-9d691fa990dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:22:54.797501 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.797466 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:54.797501 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.797503 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/85f9c559-fe10-4601-b79b-9d691fa990dc-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:54.797728 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.797513 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85f9c559-fe10-4601-b79b-9d691fa990dc-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:54.797728 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:54.797523 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rsrn\" (UniqueName: \"kubernetes.io/projected/85f9c559-fe10-4601-b79b-9d691fa990dc-kube-api-access-6rsrn\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:55.076637 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:55.076602 4399 generic.go:356] "Generic (PLEG): container finished" podID="85f9c559-fe10-4601-b79b-9d691fa990dc" containerID="6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3" exitCode=0 Oct 31 15:22:55.076831 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:55.076650 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" Oct 31 15:22:55.076831 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:55.076690 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" event={"ID":"85f9c559-fe10-4601-b79b-9d691fa990dc","Type":"ContainerDied","Data":"6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3"} Oct 31 15:22:55.076831 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:55.076762 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" event={"ID":"85f9c559-fe10-4601-b79b-9d691fa990dc","Type":"ContainerDied","Data":"1dc56d2cd0b7885cb6fc8a5e8d359c6f7583ddb6c231336a74607764fdf92eda"} Oct 31 15:22:55.076831 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:55.076792 4399 scope.go:122] "RemoveContainer" containerID="6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3" Oct 31 15:22:55.086584 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:55.086567 4399 scope.go:122] "RemoveContainer" containerID="6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3" Oct 31 15:22:55.086899 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:22:55.086873 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3\": container with ID starting with 6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3 not found: ID does not exist" containerID="6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3" Oct 31 15:22:55.086959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:55.086908 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3"} err="failed to get container status \"6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3\": rpc error: code = NotFound desc = could not find container \"6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3\": container with ID starting with 6573a5d0aa0f3a5dd5291f4da5e2e911069273e98b316e0181e382cdaac4acd3 not found: ID does not exist" Oct 31 15:22:55.250729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:55.250617 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5"] Oct 31 15:22:55.252357 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:55.252333 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5"] Oct 31 15:22:56.207251 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:22:56.207217 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:22:56.896748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:56.896717 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f9c559-fe10-4601-b79b-9d691fa990dc" path="/var/lib/kubelet/pods/85f9c559-fe10-4601-b79b-9d691fa990dc/volumes" Oct 31 15:22:57.547332 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:57.547298 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-379d72-6b44f5d67f-64bw5" podUID="85f9c559-fe10-4601-b79b-9d691fa990dc" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.110:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:22:58.781494 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.781439 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:22:58.843612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.843584 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-kube-api-access-flkp6\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-kube-api-access-flkp6\") pod \"75f8748b-f6c8-4581-942e-e2a6524d6e97\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " Oct 31 15:22:58.843612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.843617 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-token\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-token\") pod \"75f8748b-f6c8-4581-942e-e2a6524d6e97\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " Oct 31 15:22:58.843876 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.843665 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-podinfo\") pod \"75f8748b-f6c8-4581-942e-e2a6524d6e97\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " Oct 31 15:22:58.843876 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.843689 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-credential-socket\") pod \"75f8748b-f6c8-4581-942e-e2a6524d6e97\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " Oct 31 15:22:58.843876 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.843763 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-envoy\") pod \"75f8748b-f6c8-4581-942e-e2a6524d6e97\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " Oct 31 15:22:58.843876 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.843861 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-data\") pod \"75f8748b-f6c8-4581-942e-e2a6524d6e97\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " Oct 31 15:22:58.844081 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.843907 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-certs\") pod \"75f8748b-f6c8-4581-942e-e2a6524d6e97\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " Oct 31 15:22:58.844081 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.844006 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-socket\") pod \"75f8748b-f6c8-4581-942e-e2a6524d6e97\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " Oct 31 15:22:58.844081 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.844059 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75f8748b-f6c8-4581-942e-e2a6524d6e97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f8748b-f6c8-4581-942e-e2a6524d6e97-istiod-ca-cert\") pod \"75f8748b-f6c8-4581-942e-e2a6524d6e97\" (UID: \"75f8748b-f6c8-4581-942e-e2a6524d6e97\") " Oct 31 15:22:58.845179 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.844821 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-credential-socket" pod "75f8748b-f6c8-4581-942e-e2a6524d6e97" (UID: "75f8748b-f6c8-4581-942e-e2a6524d6e97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:22:58.845179 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.845049 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-certs" pod "75f8748b-f6c8-4581-942e-e2a6524d6e97" (UID: "75f8748b-f6c8-4581-942e-e2a6524d6e97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:22:58.845179 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.845112 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-data" pod "75f8748b-f6c8-4581-942e-e2a6524d6e97" (UID: "75f8748b-f6c8-4581-942e-e2a6524d6e97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:22:58.845179 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.845153 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-socket" pod "75f8748b-f6c8-4581-942e-e2a6524d6e97" (UID: "75f8748b-f6c8-4581-942e-e2a6524d6e97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:22:58.845458 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.845259 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75f8748b-f6c8-4581-942e-e2a6524d6e97-istiod-ca-cert" pod "75f8748b-f6c8-4581-942e-e2a6524d6e97" (UID: "75f8748b-f6c8-4581-942e-e2a6524d6e97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:22:58.846239 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.846046 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-podinfo" pod "75f8748b-f6c8-4581-942e-e2a6524d6e97" (UID: "75f8748b-f6c8-4581-942e-e2a6524d6e97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:22:58.846504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.846458 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-envoy" pod "75f8748b-f6c8-4581-942e-e2a6524d6e97" (UID: "75f8748b-f6c8-4581-942e-e2a6524d6e97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:22:58.846585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.846551 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-kube-api-access-flkp6" pod "75f8748b-f6c8-4581-942e-e2a6524d6e97" (UID: "75f8748b-f6c8-4581-942e-e2a6524d6e97"). InnerVolumeSpecName "kube-api-access-flkp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:22:58.846671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.846649 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-token" pod "75f8748b-f6c8-4581-942e-e2a6524d6e97" (UID: "75f8748b-f6c8-4581-942e-e2a6524d6e97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:22:58.946144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.946107 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:58.946144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.946134 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:58.946144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.946144 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75f8748b-f6c8-4581-942e-e2a6524d6e97-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:58.946144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.946153 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flkp6\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-kube-api-access-flkp6\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:58.946431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.946162 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:58.946431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.946172 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:58.946431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.946180 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:58.946431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.946189 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:58.946431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:58.946196 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75f8748b-f6c8-4581-942e-e2a6524d6e97-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:22:59.100811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.100617 4399 generic.go:356] "Generic (PLEG): container finished" podID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerID="bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6" exitCode=0 Oct 31 15:22:59.100811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.100720 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" Oct 31 15:22:59.100811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.100772 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" event={"ID":"75f8748b-f6c8-4581-942e-e2a6524d6e97","Type":"ContainerDied","Data":"bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6"} Oct 31 15:22:59.101068 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.100914 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh" event={"ID":"75f8748b-f6c8-4581-942e-e2a6524d6e97","Type":"ContainerDied","Data":"4e49a8aa32afd845f2e18a5241acb4034bbb0769bb9df65214f04259510d2463"} Oct 31 15:22:59.101068 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.100946 4399 scope.go:122] "RemoveContainer" containerID="bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6" Oct 31 15:22:59.110887 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.110856 4399 scope.go:122] "RemoveContainer" containerID="9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf" Oct 31 15:22:59.119635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.119607 4399 scope.go:122] "RemoveContainer" containerID="bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6" Oct 31 15:22:59.119915 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:22:59.119896 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6\": container with ID starting with bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6 not found: ID does not exist" containerID="bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6" Oct 31 15:22:59.119984 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.119921 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6"} err="failed to get container status \"bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6\": rpc error: code = NotFound desc = could not find container \"bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6\": container with ID starting with bcb6de4e4b86df7824f9c9502e8b2276ceb81c2add994fa8fc067bc21f4c93a6 not found: ID does not exist" Oct 31 15:22:59.119984 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.119939 4399 scope.go:122] "RemoveContainer" containerID="9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf" Oct 31 15:22:59.120191 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:22:59.120169 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf\": container with ID starting with 9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf not found: ID does not exist" containerID="9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf" Oct 31 15:22:59.120238 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.120202 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf"} err="failed to get container status \"9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf\": rpc error: code = NotFound desc = could not find container \"9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf\": container with ID starting with 9127c0362eba0ec27fe00978da914ef8d861cb81ef92b5d30a1629c0d921e4cf not found: ID does not exist" Oct 31 15:22:59.263509 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.263479 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh"] Oct 31 15:22:59.269822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:22:59.269797 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b62069-openshift-default-b48dfb5df-fthjh"] Oct 31 15:23:00.896216 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:00.896186 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" path="/var/lib/kubelet/pods/75f8748b-f6c8-4581-942e-e2a6524d6e97/volumes" Oct 31 15:23:05.833205 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:05.833174 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj"] Oct 31 15:23:05.833735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:05.833383 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" podUID="8f67d7b4-0b14-4390-bc24-58ce77918f57" containerName="maas-api" containerID="cri-o://95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c" gracePeriod=30 Oct 31 15:23:06.128738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.127495 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:23:06.139764 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.139737 4399 generic.go:356] "Generic (PLEG): container finished" podID="8f67d7b4-0b14-4390-bc24-58ce77918f57" containerID="95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c" exitCode=0 Oct 31 15:23:06.139908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.139780 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" Oct 31 15:23:06.139908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.139786 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" event={"ID":"8f67d7b4-0b14-4390-bc24-58ce77918f57","Type":"ContainerDied","Data":"95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c"} Oct 31 15:23:06.139908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.139827 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj" event={"ID":"8f67d7b4-0b14-4390-bc24-58ce77918f57","Type":"ContainerDied","Data":"841430fbddbf471fba26943dba15317e3eae404d91a25df5bb67e0cc2190eca2"} Oct 31 15:23:06.139908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.139849 4399 scope.go:122] "RemoveContainer" containerID="95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c" Oct 31 15:23:06.149115 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.149094 4399 scope.go:122] "RemoveContainer" containerID="95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c" Oct 31 15:23:06.149359 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:23:06.149342 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c\": container with ID starting with 95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c not found: ID does not exist" containerID="95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c" Oct 31 15:23:06.149406 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.149367 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c"} err="failed to get container status \"95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c\": rpc error: code = NotFound desc = could not find container \"95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c\": container with ID starting with 95ca7d86fb894d4e078b0eb5b233fba137bd40a1097f6b262293f2ec73b4310c not found: ID does not exist" Oct 31 15:23:06.232178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.232149 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-metrics-certs\") pod \"8f67d7b4-0b14-4390-bc24-58ce77918f57\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " Oct 31 15:23:06.232338 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.232235 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls\") pod \"8f67d7b4-0b14-4390-bc24-58ce77918f57\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " Oct 31 15:23:06.232338 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.232275 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f67d7b4-0b14-4390-bc24-58ce77918f57-kube-api-access-wmhgq\" (UniqueName: \"kubernetes.io/projected/8f67d7b4-0b14-4390-bc24-58ce77918f57-kube-api-access-wmhgq\") pod \"8f67d7b4-0b14-4390-bc24-58ce77918f57\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " Oct 31 15:23:06.232481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.232428 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f67d7b4-0b14-4390-bc24-58ce77918f57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f67d7b4-0b14-4390-bc24-58ce77918f57-openshift-service-ca\") pod \"8f67d7b4-0b14-4390-bc24-58ce77918f57\" (UID: \"8f67d7b4-0b14-4390-bc24-58ce77918f57\") " Oct 31 15:23:06.232905 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.232871 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f67d7b4-0b14-4390-bc24-58ce77918f57-openshift-service-ca" pod "8f67d7b4-0b14-4390-bc24-58ce77918f57" (UID: "8f67d7b4-0b14-4390-bc24-58ce77918f57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:23:06.234667 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.234645 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls" pod "8f67d7b4-0b14-4390-bc24-58ce77918f57" (UID: "8f67d7b4-0b14-4390-bc24-58ce77918f57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:23:06.234782 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.234663 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-metrics-certs" pod "8f67d7b4-0b14-4390-bc24-58ce77918f57" (UID: "8f67d7b4-0b14-4390-bc24-58ce77918f57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:23:06.234782 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.234736 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f67d7b4-0b14-4390-bc24-58ce77918f57-kube-api-access-wmhgq" pod "8f67d7b4-0b14-4390-bc24-58ce77918f57" (UID: "8f67d7b4-0b14-4390-bc24-58ce77918f57"). InnerVolumeSpecName "kube-api-access-wmhgq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:23:06.333422 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.333393 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:06.333422 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.333421 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f67d7b4-0b14-4390-bc24-58ce77918f57-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:06.333617 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.333437 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhgq\" (UniqueName: \"kubernetes.io/projected/8f67d7b4-0b14-4390-bc24-58ce77918f57-kube-api-access-wmhgq\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:06.333617 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.333451 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f67d7b4-0b14-4390-bc24-58ce77918f57-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:23:06.560255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.560219 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj"] Oct 31 15:23:06.562599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.562575 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-15cc44-84c8dd9758-tbclj"] Oct 31 15:23:06.896763 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:06.896667 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f67d7b4-0b14-4390-bc24-58ce77918f57" path="/var/lib/kubelet/pods/8f67d7b4-0b14-4390-bc24-58ce77918f57/volumes" Oct 31 15:23:12.601370 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.601335 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh"] Oct 31 15:23:12.601928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.601913 4399 cpu_manager.go:414] "containerMap: removing container" podUID="0d406da3-3311-47a8-b443-679c059505a6" containerName="maas-api" Oct 31 15:23:12.601979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.601930 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d406da3-3311-47a8-b443-679c059505a6" containerName="maas-api" Oct 31 15:23:12.601979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.601940 4399 cpu_manager.go:414] "containerMap: removing container" podUID="85f9c559-fe10-4601-b79b-9d691fa990dc" containerName="maas-api" Oct 31 15:23:12.601979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.601945 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f9c559-fe10-4601-b79b-9d691fa990dc" containerName="maas-api" Oct 31 15:23:12.601979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.601956 4399 cpu_manager.go:414] "containerMap: removing container" podUID="8f67d7b4-0b14-4390-bc24-58ce77918f57" containerName="maas-api" Oct 31 15:23:12.601979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.601962 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f67d7b4-0b14-4390-bc24-58ce77918f57" containerName="maas-api" Oct 31 15:23:12.601979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.601969 4399 cpu_manager.go:414] "containerMap: removing container" podUID="ae8969a7-0130-4522-8955-549f2e16ae15" containerName="maas-api" Oct 31 15:23:12.601979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.601974 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8969a7-0130-4522-8955-549f2e16ae15" containerName="maas-api" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602003 4399 cpu_manager.go:414] "containerMap: removing container" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602009 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602015 4399 cpu_manager.go:414] "containerMap: removing container" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602020 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602096 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602104 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75f8748b-f6c8-4581-942e-e2a6524d6e97" containerName="istio-proxy" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602113 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d406da3-3311-47a8-b443-679c059505a6" containerName="maas-api" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602119 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f9c559-fe10-4601-b79b-9d691fa990dc" containerName="maas-api" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602125 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f67d7b4-0b14-4390-bc24-58ce77918f57" containerName="maas-api" Oct 31 15:23:12.602195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.602131 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8969a7-0130-4522-8955-549f2e16ae15" containerName="maas-api" Oct 31 15:23:12.607185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.607167 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.699281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.699235 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ck2w\" (UniqueName: \"kubernetes.io/projected/bfc30e0b-d599-4bdf-a30e-dec18a791056-kube-api-access-9ck2w\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.699449 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.699309 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.699606 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.699577 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc30e0b-d599-4bdf-a30e-dec18a791056-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.699685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.699640 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-metrics-certs\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.709821 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.709795 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-ebadea-serving-cert\"" Oct 31 15:23:12.712500 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.712482 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh"] Oct 31 15:23:12.800513 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.800478 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc30e0b-d599-4bdf-a30e-dec18a791056-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.800671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.800538 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-metrics-certs\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.800671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.800633 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ck2w\" (UniqueName: \"kubernetes.io/projected/bfc30e0b-d599-4bdf-a30e-dec18a791056-kube-api-access-9ck2w\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.800841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.800672 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.800956 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:23:12.800932 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-serving-cert: secret "maas-api-e2e-praxis-b-ebadea-serving-cert" not found Oct 31 15:23:12.801024 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:23:12.801003 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls podName:bfc30e0b-d599-4bdf-a30e-dec18a791056 nodeName:}" failed. No retries permitted until 2026-10-31 15:23:13.300983713 +0000 UTC m=+2236.512225107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls") pod "maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" (UID: "bfc30e0b-d599-4bdf-a30e-dec18a791056") : secret "maas-api-e2e-praxis-b-ebadea-serving-cert" not found Oct 31 15:23:12.801294 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.801273 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc30e0b-d599-4bdf-a30e-dec18a791056-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.803314 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.803286 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-metrics-certs\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:12.857998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:12.857930 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ck2w\" (UniqueName: \"kubernetes.io/projected/bfc30e0b-d599-4bdf-a30e-dec18a791056-kube-api-access-9ck2w\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:13.307655 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:13.307621 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:13.310396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:13.310370 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls\") pod \"maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:13.520396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:13.520356 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:13.911045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:13.911019 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh"] Oct 31 15:23:13.913533 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:23:13.913507 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc30e0b_d599_4bdf_a30e_dec18a791056.slice/crio-de139770e919b2b28f6ca9c1541a08310460884f21a2213c24b69ca50c1bfb5a WatchSource:0}: Error finding container de139770e919b2b28f6ca9c1541a08310460884f21a2213c24b69ca50c1bfb5a: Status 404 returned error can't find the container with id de139770e919b2b28f6ca9c1541a08310460884f21a2213c24b69ca50c1bfb5a Oct 31 15:23:13.914946 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:13.914928 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 15:23:14.186155 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:14.186064 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" event={"ID":"bfc30e0b-d599-4bdf-a30e-dec18a791056","Type":"ContainerStarted","Data":"de139770e919b2b28f6ca9c1541a08310460884f21a2213c24b69ca50c1bfb5a"} Oct 31 15:23:15.193433 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:15.193346 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" event={"ID":"bfc30e0b-d599-4bdf-a30e-dec18a791056","Type":"ContainerStarted","Data":"ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5"} Oct 31 15:23:15.193836 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:15.193484 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:15.307999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:15.307947 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" podStartSLOduration=2.3784316690000002 podStartE2EDuration="3.307933764s" totalImagesPullingTime="929.502095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:15.306378473 +0000 UTC m=+2238.517619886" watchObservedRunningTime="2026-10-31 15:23:15.307933764 +0000 UTC m=+2238.519175175" Oct 31 15:23:21.206063 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:21.206036 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:23:26.206780 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:23:26.206741 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:23:50.012714 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.012673 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk"] Oct 31 15:23:50.016974 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.016957 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.120110 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.120075 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bd956c-openshift-default-dockercfg-gp6f6\"" Oct 31 15:23:50.122739 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.122715 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk"] Oct 31 15:23:50.134353 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.134325 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-token\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.134579 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.134549 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-podinfo\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.134765 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.134604 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-socket\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.135333 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.135307 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-envoy\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.135576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.135536 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istiod-ca-cert\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.135576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.135574 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-certs\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.135837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.135618 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-data\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.135837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.135653 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-credential-socket\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.135837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.135761 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2trrk\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-kube-api-access-2trrk\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237034 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.236991 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-token\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237040 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-podinfo\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237075 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-socket\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237130 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-envoy\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237190 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istiod-ca-cert\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237215 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-certs\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237258 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-data\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237298 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-credential-socket\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237407 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2trrk\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-kube-api-access-2trrk\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237656 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237541 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-socket\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237763 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237736 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-certs\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237739 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-data\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.237952 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237930 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-credential-socket\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.238002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.237961 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istiod-ca-cert\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.239487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.239470 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-envoy\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.239787 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.239769 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-podinfo\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.294189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.294115 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-token\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.295867 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.295836 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2trrk\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-kube-api-access-2trrk\") pod \"e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.330487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.330456 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:50.758774 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.758750 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk"] Oct 31 15:23:50.760944 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:23:50.760914 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07de2d1_ed86_42d3_9da3_2f38a7eea553.slice/crio-ad65091adb24548f23e4a79f3d065be8c781b8f3bd0fe81fe675011075e1bf64 WatchSource:0}: Error finding container ad65091adb24548f23e4a79f3d065be8c781b8f3bd0fe81fe675011075e1bf64: Status 404 returned error can't find the container with id ad65091adb24548f23e4a79f3d065be8c781b8f3bd0fe81fe675011075e1bf64 Oct 31 15:23:50.763544 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.763514 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:23:50.763599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.763578 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:23:50.763635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:50.763608 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:23:51.421487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:51.421408 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" event={"ID":"e07de2d1-ed86-42d3-9da3-2f38a7eea553","Type":"ContainerStarted","Data":"95da7c94907847e5792c2cd328016380cfb5259377614fe59cf01b23e7f658d1"} Oct 31 15:23:51.421487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:51.421469 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" event={"ID":"e07de2d1-ed86-42d3-9da3-2f38a7eea553","Type":"ContainerStarted","Data":"ad65091adb24548f23e4a79f3d065be8c781b8f3bd0fe81fe675011075e1bf64"} Oct 31 15:23:51.540497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:51.540445 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" podStartSLOduration=2.540431412 podStartE2EDuration="2.540431412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:51.538047447 +0000 UTC m=+2274.749288895" watchObservedRunningTime="2026-10-31 15:23:51.540431412 +0000 UTC m=+2274.751672820" Oct 31 15:23:52.330648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:52.330621 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:52.336334 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:52.336311 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:52.427189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:52.427160 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:52.428056 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:52.428039 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:23:53.781223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.781186 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2"] Oct 31 15:23:53.786280 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.786255 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.831825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.831790 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm"] Oct 31 15:23:53.836659 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.836636 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.843327 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.842763 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2348-openshift-default-dockercfg-k9qcg\"" Oct 31 15:23:53.880888 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.880844 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rc9p\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-kube-api-access-2rc9p\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.881053 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.880955 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.881053 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.880998 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.881053 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.881041 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.881229 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.881133 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.881229 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.881174 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.881229 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.881209 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.881365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.881245 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4683bd12-f3a9-48a0-a120-c701c479d54c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.881365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.881331 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.889855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.888520 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2"] Oct 31 15:23:53.918381 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.918353 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm"] Oct 31 15:23:53.985439 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.985398 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.990810 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.990776 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.991046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.991027 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070c7696-f269-4704-949a-93c33f3aac79-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.991197 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.991182 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.991316 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.991302 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.992748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.991418 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.992880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.992787 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.992880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.992687 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.992880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.992826 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4683bd12-f3a9-48a0-a120-c701c479d54c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.993220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.992886 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.993220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.992931 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65jvr\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-kube-api-access-65jvr\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.993220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.992980 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.993220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993020 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.993220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993045 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070c7696-f269-4704-949a-93c33f3aac79-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.993220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993120 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.993220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993172 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rc9p\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-kube-api-access-2rc9p\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.993609 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993227 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:53.993609 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993286 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.993609 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993307 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.993609 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993477 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4683bd12-f3a9-48a0-a120-c701c479d54c-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.993609 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993577 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.993894 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.993785 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.994077 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.994057 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.994986 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.994960 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:53.996635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:53.996612 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:54.049170 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.049102 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:54.049847 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.049828 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rc9p\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-kube-api-access-2rc9p\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:54.094710 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.094665 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070c7696-f269-4704-949a-93c33f3aac79-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.094852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.094748 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.094852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.094787 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.094852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.094820 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65jvr\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-kube-api-access-65jvr\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.094891 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.094913 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070c7696-f269-4704-949a-93c33f3aac79-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.094973 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.095048 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.095104 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095295 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.095270 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095444 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.095418 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095554 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.095447 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095554 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.095477 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.095648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.095595 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070c7696-f269-4704-949a-93c33f3aac79-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.097194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.097170 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070c7696-f269-4704-949a-93c33f3aac79-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.097380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.097363 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.101477 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.101455 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:54.172722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.153068 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65jvr\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-kube-api-access-65jvr\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.172722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.162193 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.211232 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.210939 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:54.563569 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.563535 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2"] Oct 31 15:23:54.604266 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:23:54.604234 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4683bd12_f3a9_48a0_a120_c701c479d54c.slice/crio-e22db0503aca7f21a2cd6a7172b2b4754601f3a34f348448a1139b8950313002 WatchSource:0}: Error finding container e22db0503aca7f21a2cd6a7172b2b4754601f3a34f348448a1139b8950313002: Status 404 returned error can't find the container with id e22db0503aca7f21a2cd6a7172b2b4754601f3a34f348448a1139b8950313002 Oct 31 15:23:54.606827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.606794 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:23:54.606900 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.606874 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:23:54.606940 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.606903 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:23:54.616426 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.616390 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm"] Oct 31 15:23:54.620276 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:23:54.620243 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070c7696_f269_4704_949a_93c33f3aac79.slice/crio-a53e133b046a92fb744047738ba346e945d52135efcdbb57a327055cc0125061 WatchSource:0}: Error finding container a53e133b046a92fb744047738ba346e945d52135efcdbb57a327055cc0125061: Status 404 returned error can't find the container with id a53e133b046a92fb744047738ba346e945d52135efcdbb57a327055cc0125061 Oct 31 15:23:54.623165 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.623122 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:23:54.623256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.623228 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:23:54.623304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:54.623278 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:23:55.445666 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:55.445632 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" event={"ID":"070c7696-f269-4704-949a-93c33f3aac79","Type":"ContainerStarted","Data":"a40cabb34c0447d7581d507ce55f166601f3a8279c45066e8b7ffcb3a56d827f"} Oct 31 15:23:55.446578 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:55.445683 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" event={"ID":"070c7696-f269-4704-949a-93c33f3aac79","Type":"ContainerStarted","Data":"a53e133b046a92fb744047738ba346e945d52135efcdbb57a327055cc0125061"} Oct 31 15:23:55.447461 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:55.447426 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" event={"ID":"4683bd12-f3a9-48a0-a120-c701c479d54c","Type":"ContainerStarted","Data":"13c1b5a6a66e977d3f38b288d73de52c1a7a5000f94796c89a2fff6db73fb6cf"} Oct 31 15:23:55.447590 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:55.447478 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" event={"ID":"4683bd12-f3a9-48a0-a120-c701c479d54c","Type":"ContainerStarted","Data":"e22db0503aca7f21a2cd6a7172b2b4754601f3a34f348448a1139b8950313002"} Oct 31 15:23:55.568732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:55.567836 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podStartSLOduration=2.567815296 podStartE2EDuration="2.567815296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:55.566747748 +0000 UTC m=+2278.777989236" watchObservedRunningTime="2026-10-31 15:23:55.567815296 +0000 UTC m=+2278.779056709" Oct 31 15:23:55.693845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:55.693782 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podStartSLOduration=2.693761985 podStartE2EDuration="2.693761985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:23:55.690281125 +0000 UTC m=+2278.901522561" watchObservedRunningTime="2026-10-31 15:23:55.693761985 +0000 UTC m=+2278.905003402" Oct 31 15:23:56.102297 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:56.102269 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:23:56.103565 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:56.103541 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 15:23:56.103798 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:56.103776 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 15:23:56.215241 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:56.215207 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:23:56.219844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:56.219814 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 15:23:56.220919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:56.220887 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 15:23:56.310282 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:23:56.310232 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:23:57.103675 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:57.103619 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 15:23:57.104191 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:57.103687 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 15:23:57.212519 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:57.212482 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 15:23:57.212735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:57.212532 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 15:23:58.102231 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:58.102197 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 15:23:58.102431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:58.102249 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 15:23:58.212054 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:58.212020 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 15:23:58.212466 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:58.212074 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 15:23:59.104038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:59.103996 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 15:23:59.104263 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:59.104046 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 15:23:59.212864 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:59.212824 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 15:23:59.213366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:23:59.212880 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 15:24:00.115724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:00.113868 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 15:24:00.115724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:00.113927 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 15:24:00.219983 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:00.217116 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 15:24:00.219983 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:00.217179 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 15:24:01.080722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:01.079825 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82"] Oct 31 15:24:01.080722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:01.080062 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" containerID="cri-o://0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6" gracePeriod=30 Oct 31 15:24:01.102427 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:01.102332 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 15:24:01.102427 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:01.102387 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 15:24:01.212014 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:01.211977 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 15:24:01.212175 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:01.212045 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 15:24:02.102436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:02.102401 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 31 15:24:02.102832 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:02.102457 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 31 15:24:02.212027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:02.211991 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Oct 31 15:24:02.212188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:02.212060 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Oct 31 15:24:02.643913 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:02.643866 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:24:02.644121 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:02.643934 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:24:04.135570 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:04.135533 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr"] Oct 31 15:24:04.136141 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:04.135761 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" containerID="cri-o://a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca" gracePeriod=30 Oct 31 15:24:04.144368 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:04.143826 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:24:04.144368 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:04.143883 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:24:04.252872 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:04.252828 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:24:04.253082 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:04.252889 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:24:05.143176 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:05.143146 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:24:05.144817 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:05.144794 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:24:05.185894 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:05.185851 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:24:05.186087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:05.185918 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:24:05.271110 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:05.271077 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:24:05.464789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:05.464754 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:24:05.465151 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:05.465116 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:24:05.468615 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:05.468593 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:24:06.348592 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.348539 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:24:06.432315 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432284 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-kube-api-access-s8gn2\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-kube-api-access-s8gn2\") pod \"a8461d1f-5523-47e3-a74f-9ceae9650c08\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " Oct 31 15:24:06.432475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432331 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-credential-socket\") pod \"a8461d1f-5523-47e3-a74f-9ceae9650c08\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " Oct 31 15:24:06.432475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432362 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-token\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-token\") pod \"a8461d1f-5523-47e3-a74f-9ceae9650c08\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " Oct 31 15:24:06.432475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432429 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-certs\") pod \"a8461d1f-5523-47e3-a74f-9ceae9650c08\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " Oct 31 15:24:06.432591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432474 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-socket\") pod \"a8461d1f-5523-47e3-a74f-9ceae9650c08\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " Oct 31 15:24:06.432645 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432612 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-data\") pod \"a8461d1f-5523-47e3-a74f-9ceae9650c08\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " Oct 31 15:24:06.432735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432643 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-credential-socket" pod "a8461d1f-5523-47e3-a74f-9ceae9650c08" (UID: "a8461d1f-5523-47e3-a74f-9ceae9650c08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:06.432735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432670 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-envoy\") pod \"a8461d1f-5523-47e3-a74f-9ceae9650c08\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " Oct 31 15:24:06.432852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432760 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-certs" pod "a8461d1f-5523-47e3-a74f-9ceae9650c08" (UID: "a8461d1f-5523-47e3-a74f-9ceae9650c08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:06.432852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432807 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8461d1f-5523-47e3-a74f-9ceae9650c08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8461d1f-5523-47e3-a74f-9ceae9650c08-istiod-ca-cert\") pod \"a8461d1f-5523-47e3-a74f-9ceae9650c08\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " Oct 31 15:24:06.432852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432845 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-podinfo\") pod \"a8461d1f-5523-47e3-a74f-9ceae9650c08\" (UID: \"a8461d1f-5523-47e3-a74f-9ceae9650c08\") " Oct 31 15:24:06.433019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432840 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-socket" pod "a8461d1f-5523-47e3-a74f-9ceae9650c08" (UID: "a8461d1f-5523-47e3-a74f-9ceae9650c08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:06.433019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.432866 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-data" pod "a8461d1f-5523-47e3-a74f-9ceae9650c08" (UID: "a8461d1f-5523-47e3-a74f-9ceae9650c08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:06.433128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.433098 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8461d1f-5523-47e3-a74f-9ceae9650c08-istiod-ca-cert" pod "a8461d1f-5523-47e3-a74f-9ceae9650c08" (UID: "a8461d1f-5523-47e3-a74f-9ceae9650c08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:24:06.433596 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.433567 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:06.433747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.433598 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8461d1f-5523-47e3-a74f-9ceae9650c08-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:06.433747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.433614 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:06.433747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.433628 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:06.433747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.433644 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:06.434716 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.434680 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-kube-api-access-s8gn2" pod "a8461d1f-5523-47e3-a74f-9ceae9650c08" (UID: "a8461d1f-5523-47e3-a74f-9ceae9650c08"). InnerVolumeSpecName "kube-api-access-s8gn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:06.434869 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.434852 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-envoy" pod "a8461d1f-5523-47e3-a74f-9ceae9650c08" (UID: "a8461d1f-5523-47e3-a74f-9ceae9650c08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:06.434927 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.434879 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-token" pod "a8461d1f-5523-47e3-a74f-9ceae9650c08" (UID: "a8461d1f-5523-47e3-a74f-9ceae9650c08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:06.435299 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.435274 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-podinfo" pod "a8461d1f-5523-47e3-a74f-9ceae9650c08" (UID: "a8461d1f-5523-47e3-a74f-9ceae9650c08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:24:06.535038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.535003 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:06.535038 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.535036 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:06.535273 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.535048 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8gn2\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-kube-api-access-s8gn2\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:06.535273 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.535058 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8461d1f-5523-47e3-a74f-9ceae9650c08-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:06.591151 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.591120 4399 generic.go:356] "Generic (PLEG): container finished" podID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerID="0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6" exitCode=0 Oct 31 15:24:06.591305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.591170 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" Oct 31 15:24:06.591305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.591202 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" event={"ID":"a8461d1f-5523-47e3-a74f-9ceae9650c08","Type":"ContainerDied","Data":"0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6"} Oct 31 15:24:06.591305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.591258 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82" event={"ID":"a8461d1f-5523-47e3-a74f-9ceae9650c08","Type":"ContainerDied","Data":"05bd0745ebce25c7d9408620653fb03c3eab15b48a3f266932e57922697dbccd"} Oct 31 15:24:06.591305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.591287 4399 scope.go:122] "RemoveContainer" containerID="0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6" Oct 31 15:24:06.601319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.601300 4399 scope.go:122] "RemoveContainer" containerID="0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6" Oct 31 15:24:06.601607 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:24:06.601586 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6\": container with ID starting with 0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6 not found: ID does not exist" containerID="0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6" Oct 31 15:24:06.601648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.601617 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6"} err="failed to get container status \"0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6\": rpc error: code = NotFound desc = could not find container \"0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6\": container with ID starting with 0d8c28d8cb8dd249e03604dfbff2c267bb279c5766d042c167cd8ba4dfae54a6 not found: ID does not exist" Oct 31 15:24:06.753815 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.753772 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82"] Oct 31 15:24:06.760277 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.760244 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-hhc82"] Oct 31 15:24:06.900221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:06.900142 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" path="/var/lib/kubelet/pods/a8461d1f-5523-47e3-a74f-9ceae9650c08/volumes" Oct 31 15:24:07.781421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:07.781345 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-bcmwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:24:07.781421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:07.781399 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:24:08.903737 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.903690 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz"] Oct 31 15:24:08.904437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.904409 4399 cpu_manager.go:414] "containerMap: removing container" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" Oct 31 15:24:08.904493 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.904439 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" Oct 31 15:24:08.904548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.904539 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8461d1f-5523-47e3-a74f-9ceae9650c08" containerName="istio-proxy" Oct 31 15:24:08.909245 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.909226 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:08.952424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.952395 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z"] Oct 31 15:24:08.956439 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.956402 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:08.963081 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.963046 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:08.963199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.963123 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a67bace-573f-47d7-9094-9c926e0e5300-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:08.963199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.963167 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:08.963310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.963255 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpbr\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-kube-api-access-kbpbr\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:08.963676 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.963655 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:08.964107 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.964088 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:08.964361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.964344 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:08.964717 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.964675 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a67bace-573f-47d7-9094-9c926e0e5300-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:08.964941 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:08.964916 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.007158 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.007128 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz"] Oct 31 15:24:09.061806 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.061774 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z"] Oct 31 15:24:09.065758 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.065727 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd02a7e-dad7-4c8c-a279-93053f852bea-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.065908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.065779 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.065908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.065805 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.065908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.065893 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.066083 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066032 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpbr\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-kube-api-access-kbpbr\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.066176 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066157 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.066466 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066442 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.066565 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066192 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.066565 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066529 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.066680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066564 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.066680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066609 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.066680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066665 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a67bace-573f-47d7-9094-9c926e0e5300-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.066859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066743 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.066859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066773 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.066964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066862 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.066964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066890 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.066964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066945 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.067114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.066975 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a67bace-573f-47d7-9094-9c926e0e5300-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.067114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.067009 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68dkh\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-kube-api-access-68dkh\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.067222 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.067154 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.067478 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.067446 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.067602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.067583 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.068164 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.068141 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a67bace-573f-47d7-9094-9c926e0e5300-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.069213 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.069183 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.069740 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.069718 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a67bace-573f-47d7-9094-9c926e0e5300-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.122088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.122065 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.122228 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.122211 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpbr\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-kube-api-access-kbpbr\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.168194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168163 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.168320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168217 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68dkh\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-kube-api-access-68dkh\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.168320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168253 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd02a7e-dad7-4c8c-a279-93053f852bea-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.168320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168284 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.168320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168308 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.168576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168462 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.168576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168498 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.168682 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168583 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.168682 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168594 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-credential-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.168839 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.168691 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.169087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.169061 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-certs\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.169730 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.169317 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-data\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.169730 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.169465 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd02a7e-dad7-4c8c-a279-93053f852bea-istiod-ca-cert\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.169730 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.169652 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-socket\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.171645 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.171622 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-podinfo\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.171943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.171921 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-envoy\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.223267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.223245 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:09.224110 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.224082 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68dkh\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-kube-api-access-68dkh\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.224220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.224178 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-token\") pod \"e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.273688 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.273480 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:09.457330 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.457299 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 15:24:09.573168 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573129 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-podinfo\") pod \"80996ff9-8abf-41eb-98f4-32de705fc1bf\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " Oct 31 15:24:09.573380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573210 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-token\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-token\") pod \"80996ff9-8abf-41eb-98f4-32de705fc1bf\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " Oct 31 15:24:09.573380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573261 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-data\") pod \"80996ff9-8abf-41eb-98f4-32de705fc1bf\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " Oct 31 15:24:09.573380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573319 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-certs\") pod \"80996ff9-8abf-41eb-98f4-32de705fc1bf\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " Oct 31 15:24:09.573380 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573362 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80996ff9-8abf-41eb-98f4-32de705fc1bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996ff9-8abf-41eb-98f4-32de705fc1bf-istiod-ca-cert\") pod \"80996ff9-8abf-41eb-98f4-32de705fc1bf\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " Oct 31 15:24:09.573639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573415 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-credential-socket\") pod \"80996ff9-8abf-41eb-98f4-32de705fc1bf\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " Oct 31 15:24:09.573639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573457 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-kube-api-access-6pg6l\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-kube-api-access-6pg6l\") pod \"80996ff9-8abf-41eb-98f4-32de705fc1bf\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " Oct 31 15:24:09.573639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573491 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-socket\") pod \"80996ff9-8abf-41eb-98f4-32de705fc1bf\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " Oct 31 15:24:09.573639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573518 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-envoy\") pod \"80996ff9-8abf-41eb-98f4-32de705fc1bf\" (UID: \"80996ff9-8abf-41eb-98f4-32de705fc1bf\") " Oct 31 15:24:09.574009 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.573923 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-certs" pod "80996ff9-8abf-41eb-98f4-32de705fc1bf" (UID: "80996ff9-8abf-41eb-98f4-32de705fc1bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:09.574095 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.574059 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:09.574641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.574289 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80996ff9-8abf-41eb-98f4-32de705fc1bf-istiod-ca-cert" pod "80996ff9-8abf-41eb-98f4-32de705fc1bf" (UID: "80996ff9-8abf-41eb-98f4-32de705fc1bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:24:09.574641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.574396 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-credential-socket" pod "80996ff9-8abf-41eb-98f4-32de705fc1bf" (UID: "80996ff9-8abf-41eb-98f4-32de705fc1bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:09.574877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.574732 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-socket" pod "80996ff9-8abf-41eb-98f4-32de705fc1bf" (UID: "80996ff9-8abf-41eb-98f4-32de705fc1bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:09.575361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.575328 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-data" pod "80996ff9-8abf-41eb-98f4-32de705fc1bf" (UID: "80996ff9-8abf-41eb-98f4-32de705fc1bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:09.576625 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.576585 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-kube-api-access-6pg6l" pod "80996ff9-8abf-41eb-98f4-32de705fc1bf" (UID: "80996ff9-8abf-41eb-98f4-32de705fc1bf"). InnerVolumeSpecName "kube-api-access-6pg6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:09.576625 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.576602 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-token" pod "80996ff9-8abf-41eb-98f4-32de705fc1bf" (UID: "80996ff9-8abf-41eb-98f4-32de705fc1bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:24:09.577018 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.576996 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-envoy" pod "80996ff9-8abf-41eb-98f4-32de705fc1bf" (UID: "80996ff9-8abf-41eb-98f4-32de705fc1bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:24:09.578109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.578081 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-podinfo" pod "80996ff9-8abf-41eb-98f4-32de705fc1bf" (UID: "80996ff9-8abf-41eb-98f4-32de705fc1bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:24:09.610620 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.610585 4399 generic.go:356] "Generic (PLEG): container finished" podID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerID="a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca" exitCode=0 Oct 31 15:24:09.610759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.610634 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" Oct 31 15:24:09.610759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.610639 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" event={"ID":"80996ff9-8abf-41eb-98f4-32de705fc1bf","Type":"ContainerDied","Data":"a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca"} Oct 31 15:24:09.610759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.610737 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr" event={"ID":"80996ff9-8abf-41eb-98f4-32de705fc1bf","Type":"ContainerDied","Data":"d7b157b72efffd7c5286ebbf24a042e1d0c2aaa068fe05ba0d76ef7bdb89ebad"} Oct 31 15:24:09.610896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.610768 4399 scope.go:122] "RemoveContainer" containerID="a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca" Oct 31 15:24:09.621455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.621436 4399 scope.go:122] "RemoveContainer" containerID="a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca" Oct 31 15:24:09.621755 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:24:09.621733 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca\": container with ID starting with a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca not found: ID does not exist" containerID="a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca" Oct 31 15:24:09.621853 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.621760 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca"} err="failed to get container status \"a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca\": rpc error: code = NotFound desc = could not find container \"a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca\": container with ID starting with a4126ce4b52d1cfdd42bf7054eecedd66d2debe6e95eeaec32189de875e47cca not found: ID does not exist" Oct 31 15:24:09.654967 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.654942 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz"] Oct 31 15:24:09.655246 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:24:09.655226 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a67bace_573f_47d7_9094_9c926e0e5300.slice/crio-75ffbbb495bf713ce416dc2c03cdb4b55575f5e66a6a50999a0f83fded354494 WatchSource:0}: Error finding container 75ffbbb495bf713ce416dc2c03cdb4b55575f5e66a6a50999a0f83fded354494: Status 404 returned error can't find the container with id 75ffbbb495bf713ce416dc2c03cdb4b55575f5e66a6a50999a0f83fded354494 Oct 31 15:24:09.657646 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.657618 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:24:09.657769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.657679 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:24:09.657769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.657737 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:24:09.675439 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.675418 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:09.675545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.675444 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:09.675545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.675459 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80996ff9-8abf-41eb-98f4-32de705fc1bf-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:09.675545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.675473 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:09.675545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.675487 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pg6l\" (UniqueName: \"kubernetes.io/projected/80996ff9-8abf-41eb-98f4-32de705fc1bf-kube-api-access-6pg6l\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:09.675545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.675501 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:09.675545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.675514 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:09.675545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.675530 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80996ff9-8abf-41eb-98f4-32de705fc1bf-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:24:09.710661 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.710631 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z"] Oct 31 15:24:09.711181 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:24:09.711153 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdd02a7e_dad7_4c8c_a279_93053f852bea.slice/crio-7f25ebf3433e4249a3440252855600b3e4a1b35051e81993029fe11121866992 WatchSource:0}: Error finding container 7f25ebf3433e4249a3440252855600b3e4a1b35051e81993029fe11121866992: Status 404 returned error can't find the container with id 7f25ebf3433e4249a3440252855600b3e4a1b35051e81993029fe11121866992 Oct 31 15:24:09.713875 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.713842 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:24:09.713965 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.713920 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:24:09.714022 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.713963 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:24:09.776126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.776099 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr"] Oct 31 15:24:09.779734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:09.779712 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-bcmwr"] Oct 31 15:24:10.619292 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:10.619255 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" event={"ID":"cdd02a7e-dad7-4c8c-a279-93053f852bea","Type":"ContainerStarted","Data":"e882fbc72914c48af8f5aa32848a3c61f244d76f71006a602e8f5d7c43766ed6"} Oct 31 15:24:10.619786 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:10.619339 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" event={"ID":"cdd02a7e-dad7-4c8c-a279-93053f852bea","Type":"ContainerStarted","Data":"7f25ebf3433e4249a3440252855600b3e4a1b35051e81993029fe11121866992"} Oct 31 15:24:10.621324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:10.620803 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" event={"ID":"9a67bace-573f-47d7-9094-9c926e0e5300","Type":"ContainerStarted","Data":"7ef9143e9369abcc72062688e24a347e44d5bc22b5c726133bfa5c4d13391012"} Oct 31 15:24:10.621324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:10.620845 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" event={"ID":"9a67bace-573f-47d7-9094-9c926e0e5300","Type":"ContainerStarted","Data":"75ffbbb495bf713ce416dc2c03cdb4b55575f5e66a6a50999a0f83fded354494"} Oct 31 15:24:10.880786 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:10.880126 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" podStartSLOduration=2.880101786 podStartE2EDuration="2.880101786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:10.756006175 +0000 UTC m=+2293.967247588" watchObservedRunningTime="2026-10-31 15:24:10.880101786 +0000 UTC m=+2294.091343200" Oct 31 15:24:10.893421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:10.893039 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" podStartSLOduration=2.893024089 podStartE2EDuration="2.893024089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:10.87840858 +0000 UTC m=+2294.089649993" watchObservedRunningTime="2026-10-31 15:24:10.893024089 +0000 UTC m=+2294.104265495" Oct 31 15:24:10.901434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:10.901405 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" path="/var/lib/kubelet/pods/80996ff9-8abf-41eb-98f4-32de705fc1bf/volumes" Oct 31 15:24:11.223455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:11.223422 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:11.225015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:11.224994 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 15:24:11.225074 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:11.225036 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 15:24:11.288690 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:11.288654 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:11.299956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:11.299916 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 15:24:11.300948 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:11.300909 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 15:24:12.224587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:12.224550 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 15:24:12.225075 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:12.224642 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 15:24:12.278722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:12.278052 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 15:24:12.278722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:12.278114 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 15:24:13.224593 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:13.224558 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 15:24:13.225050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:13.224610 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 15:24:13.298578 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:13.298540 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 15:24:13.298832 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:13.298598 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 15:24:14.223854 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:14.223818 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 15:24:14.224013 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:14.223872 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 15:24:14.280729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:14.277081 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 15:24:14.280729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:14.277146 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 15:24:15.224093 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:15.224050 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 15:24:15.224265 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:15.224108 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 15:24:15.303246 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:15.303196 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 15:24:15.303795 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:15.303264 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 15:24:16.224550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:16.224512 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 15:24:16.224792 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:16.224586 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 15:24:16.275148 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:16.275109 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 15:24:16.275314 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:16.275180 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 15:24:17.838284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:17.838241 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:17.839327 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:17.839305 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:17.840169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:17.840149 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:24:17.925604 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:17.925409 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:17.927418 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:17.926077 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:17.928388 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:17.928368 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:24:20.722272 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.722239 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz"] Oct 31 15:24:20.722896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.722877 4399 cpu_manager.go:414] "containerMap: removing container" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" Oct 31 15:24:20.723002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.722899 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" Oct 31 15:24:20.723139 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.723125 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="80996ff9-8abf-41eb-98f4-32de705fc1bf" containerName="istio-proxy" Oct 31 15:24:20.727692 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.727675 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.788085 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.788048 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz"] Oct 31 15:24:20.875114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.875082 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b21ab-9740-45db-8700-b9d5d9809d44-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.875246 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.875125 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.875246 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.875205 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.875329 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.875295 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.875362 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.875341 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b21ab-9740-45db-8700-b9d5d9809d44-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.875398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.875363 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.875398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.875387 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.875478 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.875450 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.875535 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.875523 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbqj\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-kube-api-access-9pbqj\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.976849 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.976765 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.976849 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.976808 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b21ab-9740-45db-8700-b9d5d9809d44-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.976849 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.976842 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.976874 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.976917 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.976970 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbqj\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-kube-api-access-9pbqj\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.977099 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b21ab-9740-45db-8700-b9d5d9809d44-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977371 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.977171 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977371 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.977242 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977371 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.977275 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977371 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.977352 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977576 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.977440 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.977621 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.977818 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.977799 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b21ab-9740-45db-8700-b9d5d9809d44-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.979481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.979461 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:20.979837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:20.979820 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b21ab-9740-45db-8700-b9d5d9809d44-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:21.033253 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.033227 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:21.033358 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.033281 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbqj\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-kube-api-access-9pbqj\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-lblwz\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:21.041650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.041627 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:21.471503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.471477 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz"] Oct 31 15:24:21.473222 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:24:21.473187 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod793b21ab_9740_45db_8700_b9d5d9809d44.slice/crio-50444354509a021f8e434ac5d8b4f9d3770d3d79a1553f9110d2c072a868290d WatchSource:0}: Error finding container 50444354509a021f8e434ac5d8b4f9d3770d3d79a1553f9110d2c072a868290d: Status 404 returned error can't find the container with id 50444354509a021f8e434ac5d8b4f9d3770d3d79a1553f9110d2c072a868290d Oct 31 15:24:21.475686 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.475657 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:24:21.475758 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.475744 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:24:21.475799 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.475781 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:24:21.734135 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.734049 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" event={"ID":"793b21ab-9740-45db-8700-b9d5d9809d44","Type":"ContainerStarted","Data":"6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1"} Oct 31 15:24:21.734135 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.734115 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" event={"ID":"793b21ab-9740-45db-8700-b9d5d9809d44","Type":"ContainerStarted","Data":"50444354509a021f8e434ac5d8b4f9d3770d3d79a1553f9110d2c072a868290d"} Oct 31 15:24:21.851288 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:21.851223 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podStartSLOduration=1.851204392 podStartE2EDuration="1.851204392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:21.848774615 +0000 UTC m=+2305.060016028" watchObservedRunningTime="2026-10-31 15:24:21.851204392 +0000 UTC m=+2305.062445805" Oct 31 15:24:22.042106 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:22.042012 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:22.043684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:22.043658 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:24:22.043817 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:22.043735 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 15:24:23.059722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:23.056076 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:24:23.059722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:23.056136 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 15:24:24.046052 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:24.046014 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:24:24.046338 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:24.046306 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 15:24:25.066743 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:25.066310 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:24:25.066743 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:25.066401 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 15:24:26.055206 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:26.055168 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:24:26.055516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:26.055482 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 15:24:26.207597 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:24:26.207558 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:24:27.043232 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:27.043193 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:24:27.044677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:27.044641 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 15:24:28.044724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:28.042084 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:24:28.044724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:28.042157 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 15:24:29.073763 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:29.072991 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:24:29.073763 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:29.073049 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 15:24:30.050645 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:30.050607 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Oct 31 15:24:30.051051 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:30.051001 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Oct 31 15:24:32.084858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:32.084817 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-lblwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:24:32.085476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:32.085441 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:24:32.386848 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:32.386757 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:32.387199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:32.387165 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:32.388266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:32.388243 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:24:39.871016 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:39.870982 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9"] Oct 31 15:24:39.875338 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:39.875320 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:39.928711 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:39.928668 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c4ede8-serving-cert\"" Oct 31 15:24:39.933308 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:39.933283 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9"] Oct 31 15:24:40.042095 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.042067 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3124d69-2c22-44e9-9a87-17a793270206-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.042262 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.042188 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-metrics-certs\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.042262 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.042226 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtf8r\" (UniqueName: \"kubernetes.io/projected/f3124d69-2c22-44e9-9a87-17a793270206-kube-api-access-dtf8r\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.042262 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.042260 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.143634 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.143556 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3124d69-2c22-44e9-9a87-17a793270206-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.143851 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.143800 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-metrics-certs\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.143851 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.143839 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtf8r\" (UniqueName: \"kubernetes.io/projected/f3124d69-2c22-44e9-9a87-17a793270206-kube-api-access-dtf8r\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.144031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.143881 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.144342 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.144320 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3124d69-2c22-44e9-9a87-17a793270206-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.146665 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.146641 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.146821 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.146803 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-metrics-certs\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.202791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.202760 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtf8r\" (UniqueName: \"kubernetes.io/projected/f3124d69-2c22-44e9-9a87-17a793270206-kube-api-access-dtf8r\") pod \"maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.489163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.489132 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:40.866434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.866408 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9"] Oct 31 15:24:40.868652 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:24:40.868608 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3124d69_2c22_44e9_9a87_17a793270206.slice/crio-afecf4cee02d9fa74e78925a2739a917f58536606fb11a4051b8b48302ffa94e WatchSource:0}: Error finding container afecf4cee02d9fa74e78925a2739a917f58536606fb11a4051b8b48302ffa94e: Status 404 returned error can't find the container with id afecf4cee02d9fa74e78925a2739a917f58536606fb11a4051b8b48302ffa94e Oct 31 15:24:40.953115 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:40.953084 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" event={"ID":"f3124d69-2c22-44e9-9a87-17a793270206","Type":"ContainerStarted","Data":"afecf4cee02d9fa74e78925a2739a917f58536606fb11a4051b8b48302ffa94e"} Oct 31 15:24:41.961764 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:41.961727 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" event={"ID":"f3124d69-2c22-44e9-9a87-17a793270206","Type":"ContainerStarted","Data":"07f71bab10bacecff3ada076d1c242edbc4c6cfa63177850126df58aa9562dc5"} Oct 31 15:24:41.962171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:41.961862 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:42.078531 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:42.078420 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" podStartSLOduration=2.234768717 podStartE2EDuration="3.078405899s" totalImagesPullingTime="843.637182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:42.075959699 +0000 UTC m=+2325.287201113" watchObservedRunningTime="2026-10-31 15:24:42.078405899 +0000 UTC m=+2325.289647311" Oct 31 15:24:47.972175 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:47.972141 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:24:49.477306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.477273 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n"] Oct 31 15:24:49.481583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.481565 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.537928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.537899 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n"] Oct 31 15:24:49.553911 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.553884 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-metrics-certs\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.554042 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.554023 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knqvt\" (UniqueName: \"kubernetes.io/projected/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-kube-api-access-knqvt\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.554110 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.554084 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.554208 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.554195 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.583015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.582990 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bd956c-serving-cert\"" Oct 31 15:24:49.655664 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.655637 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-metrics-certs\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.655843 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.655750 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knqvt\" (UniqueName: \"kubernetes.io/projected/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-kube-api-access-knqvt\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.655843 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.655800 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.655932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.655878 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.656019 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:24:49.656000 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-serving-cert: secret "maas-api-e2e-praxis-b-bd956c-serving-cert" not found Oct 31 15:24:49.656093 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:24:49.656079 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls podName:f8d8ca03-af52-438c-9c1a-53a28cd20ae8 nodeName:}" failed. No retries permitted until 2026-10-31 15:24:50.156058837 +0000 UTC m=+2333.367300243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls") pod "maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" (UID: "f8d8ca03-af52-438c-9c1a-53a28cd20ae8") : secret "maas-api-e2e-praxis-b-bd956c-serving-cert" not found Oct 31 15:24:49.656358 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.656337 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.658466 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.658274 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-metrics-certs\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:49.714561 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:49.714536 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knqvt\" (UniqueName: \"kubernetes.io/projected/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-kube-api-access-knqvt\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:50.162709 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:50.162674 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:50.165220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:50.165181 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:50.395188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:50.395155 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:50.773645 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:50.771199 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n"] Oct 31 15:24:51.014388 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:51.014351 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" event={"ID":"f8d8ca03-af52-438c-9c1a-53a28cd20ae8","Type":"ContainerStarted","Data":"f441ba32c73d713d795a30a72c828c063cee1ea7135aea6238c1fbc9e0d8b755"} Oct 31 15:24:52.021990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:52.021902 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" event={"ID":"f8d8ca03-af52-438c-9c1a-53a28cd20ae8","Type":"ContainerStarted","Data":"8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b"} Oct 31 15:24:52.022347 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:52.021996 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:24:52.134945 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:52.134893 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" podStartSLOduration=2.179977443 podStartE2EDuration="3.134876025s" totalImagesPullingTime="954.898582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:24:52.133857916 +0000 UTC m=+2335.345099343" watchObservedRunningTime="2026-10-31 15:24:52.134876025 +0000 UTC m=+2335.346117440" Oct 31 15:24:56.428277 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:24:56.428224 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:24:58.067574 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:24:58.067544 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:25:06.099718 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.099669 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n"] Oct 31 15:25:06.105479 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.105454 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.223636 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.223592 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n"] Oct 31 15:25:06.243587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.243543 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.243776 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.243620 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.243776 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.243753 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.243873 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.243848 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.243927 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.243895 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e074092c-d612-4e28-a501-ca34eab4a14f-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.244046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.244023 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.244114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.244068 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpbmz\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-kube-api-access-kpbmz\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.244114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.244100 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.244245 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.244185 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e074092c-d612-4e28-a501-ca34eab4a14f-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345290 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345252 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e074092c-d612-4e28-a501-ca34eab4a14f-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345460 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345313 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345460 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345336 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpbmz\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-kube-api-access-kpbmz\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345460 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345355 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345460 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345394 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e074092c-d612-4e28-a501-ca34eab4a14f-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345460 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345449 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345799 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345490 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345799 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345523 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345799 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345571 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345978 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345822 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-certs\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.345978 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.345898 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.346105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.346028 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-data\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.346166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.346117 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e074092c-d612-4e28-a501-ca34eab4a14f-istiod-ca-cert\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.346166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.346143 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-credential-socket\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.347975 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.347951 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-envoy\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.348209 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.348193 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e074092c-d612-4e28-a501-ca34eab4a14f-istio-podinfo\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.401212 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.401146 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpbmz\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-kube-api-access-kpbmz\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.401356 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.401341 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-istio-token\") pod \"e2e-worker-w3-190693-openshift-default-797d798599-d2c2n\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.419297 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.419272 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:06.847970 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.847945 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n"] Oct 31 15:25:06.849502 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:25:06.849473 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode074092c_d612_4e28_a501_ca34eab4a14f.slice/crio-ca2672bc1e362b3e56b90cd73c39991935b2375d3c75cb70b08f249e55475719 WatchSource:0}: Error finding container ca2672bc1e362b3e56b90cd73c39991935b2375d3c75cb70b08f249e55475719: Status 404 returned error can't find the container with id ca2672bc1e362b3e56b90cd73c39991935b2375d3c75cb70b08f249e55475719 Oct 31 15:25:06.852435 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.852390 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:25:06.852556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.852477 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:25:06.852556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:06.852513 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:25:07.201890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:07.201851 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" event={"ID":"e074092c-d612-4e28-a501-ca34eab4a14f","Type":"ContainerStarted","Data":"85717764815c836dd83d89c6c58adce858f3929e9d32f2daa78b3c11edd6a0fb"} Oct 31 15:25:07.202319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:07.201914 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" event={"ID":"e074092c-d612-4e28-a501-ca34eab4a14f","Type":"ContainerStarted","Data":"ca2672bc1e362b3e56b90cd73c39991935b2375d3c75cb70b08f249e55475719"} Oct 31 15:25:07.321605 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:07.321546 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podStartSLOduration=1.321528063 podStartE2EDuration="1.321528063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:25:07.318865471 +0000 UTC m=+2350.530106886" watchObservedRunningTime="2026-10-31 15:25:07.321528063 +0000 UTC m=+2350.532769475" Oct 31 15:25:07.420410 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:07.420371 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:07.421948 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:07.421920 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 15:25:07.422093 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:07.421978 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 15:25:08.420363 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:08.420333 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 15:25:08.420813 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:08.420382 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 15:25:09.420589 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:09.420554 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 15:25:09.421059 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:09.420615 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 15:25:10.419756 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:10.419723 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 15:25:10.419939 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:10.419784 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 15:25:11.420403 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:11.420357 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 15:25:11.420956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:11.420467 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 15:25:12.420503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:12.420469 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 15:25:12.420964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:12.420533 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 15:25:13.420010 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:13.419975 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 31 15:25:13.420225 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:13.420060 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 31 15:25:14.650126 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:14.650084 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:14.650872 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:14.650848 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:14.652459 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:14.652439 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:25:26.528011 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:25:26.527953 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:25:27.637729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:27.635792 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:25:27.875032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:27.874919 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:28.044724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:28.037951 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-ht48f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:25:28.044724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:28.038044 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:28.141737 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:28.139782 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:28.654890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:28.651027 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:28.704724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:28.680158 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:25:29.161032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:29.143817 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:29.430771 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:29.426339 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:25:29.430771 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:29.426424 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:25:29.430771 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:29.428609 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 15:25:35.726866 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:35.726830 4399 generic.go:356] "Generic (PLEG): container finished" podID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerID="95da7c94907847e5792c2cd328016380cfb5259377614fe59cf01b23e7f658d1" exitCode=0 Oct 31 15:25:35.727249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:35.726904 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" event={"ID":"e07de2d1-ed86-42d3-9da3-2f38a7eea553","Type":"ContainerDied","Data":"95da7c94907847e5792c2cd328016380cfb5259377614fe59cf01b23e7f658d1"} Oct 31 15:25:35.727326 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:35.727308 4399 scope.go:122] "RemoveContainer" containerID="95da7c94907847e5792c2cd328016380cfb5259377614fe59cf01b23e7f658d1" Oct 31 15:25:35.728877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:35.728853 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:25:35.728950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:35.728917 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:25:35.728950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:35.728946 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:25:36.735684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:36.735650 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" event={"ID":"e07de2d1-ed86-42d3-9da3-2f38a7eea553","Type":"ContainerStarted","Data":"78e36ab7853c6633aa83a4678a5f13f233fbf24a5d08f2c0d442c7fb33892e46"} Oct 31 15:25:37.331177 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:37.331101 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:25:37.332520 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:37.332495 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 15:25:37.332655 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:37.332541 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 15:25:37.427108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:37.427080 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:25:38.331962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:38.331923 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 15:25:38.332383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:38.331993 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 15:25:39.331855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:39.331820 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 15:25:39.332036 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:39.331884 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 15:25:40.330816 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:40.330786 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 15:25:40.331002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:40.330848 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 15:25:41.331083 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:41.331049 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" start-of-body= Oct 31 15:25:41.331476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:41.331106 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.115:15021/healthz/ready\": dial tcp 10.133.0.115:15021: connect: connection refused" Oct 31 15:25:42.335749 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:42.335685 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:25:42.336950 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:42.336932 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:25:52.158764 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:52.158738 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kzfcq_5aac9441-7f91-49f3-9472-8fcd34fec615/payload-processing/0.log" Oct 31 15:25:52.418747 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:52.418635 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-kzfcq_5aac9441-7f91-49f3-9472-8fcd34fec615/payload-processing/0.log" Oct 31 15:25:55.260727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:55.260115 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:55.851755 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:55.831579 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded" Oct 31 15:25:56.260894 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:56.260847 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": context deadline exceeded" Oct 31 15:25:56.261956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:56.261933 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:25:56.410724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:56.410528 4399 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": context deadline exceeded" Oct 31 15:25:56.434726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:56.430799 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.581781 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:25:56.577899 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:25:56.842317 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:56.842186 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:56.843289 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:56.843266 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:25:57.263872 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:57.263826 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:25:57.658585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:57.657362 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 15:25:57.668779 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:25:57.659813 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:26:14.056319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.056287 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn"] Oct 31 15:26:14.056861 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.056519 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" podUID="e6ccd829-3732-4e8a-a8c5-883e788bede6" containerName="maas-api" containerID="cri-o://25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14" gracePeriod=30 Oct 31 15:26:14.371872 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.371843 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:26:14.381461 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.381431 4399 generic.go:356] "Generic (PLEG): container finished" podID="e6ccd829-3732-4e8a-a8c5-883e788bede6" containerID="25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14" exitCode=0 Oct 31 15:26:14.381638 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.381480 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" Oct 31 15:26:14.381638 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.381516 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" event={"ID":"e6ccd829-3732-4e8a-a8c5-883e788bede6","Type":"ContainerDied","Data":"25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14"} Oct 31 15:26:14.381638 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.381582 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn" event={"ID":"e6ccd829-3732-4e8a-a8c5-883e788bede6","Type":"ContainerDied","Data":"78d44004beab28c9653b7a4830b536f4aba53fbfadf78ffb53294315760b8b40"} Oct 31 15:26:14.381638 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.381613 4399 scope.go:122] "RemoveContainer" containerID="25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14" Oct 31 15:26:14.391586 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.391561 4399 scope.go:122] "RemoveContainer" containerID="25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14" Oct 31 15:26:14.391899 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:26:14.391881 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14\": container with ID starting with 25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14 not found: ID does not exist" containerID="25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14" Oct 31 15:26:14.391961 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.391905 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14"} err="failed to get container status \"25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14\": rpc error: code = NotFound desc = could not find container \"25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14\": container with ID starting with 25dbcbacafc8698eb6dba6d46183881b6fc8f4a1140c3c7b274965346f774b14 not found: ID does not exist" Oct 31 15:26:14.397280 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.397258 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ccd829-3732-4e8a-a8c5-883e788bede6-kube-api-access-l5wzp\" (UniqueName: \"kubernetes.io/projected/e6ccd829-3732-4e8a-a8c5-883e788bede6-kube-api-access-l5wzp\") pod \"e6ccd829-3732-4e8a-a8c5-883e788bede6\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " Oct 31 15:26:14.397383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.397368 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-metrics-certs\") pod \"e6ccd829-3732-4e8a-a8c5-883e788bede6\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " Oct 31 15:26:14.397490 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.397478 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ccd829-3732-4e8a-a8c5-883e788bede6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ccd829-3732-4e8a-a8c5-883e788bede6-openshift-service-ca\") pod \"e6ccd829-3732-4e8a-a8c5-883e788bede6\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " Oct 31 15:26:14.397567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.397555 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls\") pod \"e6ccd829-3732-4e8a-a8c5-883e788bede6\" (UID: \"e6ccd829-3732-4e8a-a8c5-883e788bede6\") " Oct 31 15:26:14.397822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.397803 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ccd829-3732-4e8a-a8c5-883e788bede6-openshift-service-ca" pod "e6ccd829-3732-4e8a-a8c5-883e788bede6" (UID: "e6ccd829-3732-4e8a-a8c5-883e788bede6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:14.397920 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.397900 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ccd829-3732-4e8a-a8c5-883e788bede6-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:14.399475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.399455 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ccd829-3732-4e8a-a8c5-883e788bede6-kube-api-access-l5wzp" pod "e6ccd829-3732-4e8a-a8c5-883e788bede6" (UID: "e6ccd829-3732-4e8a-a8c5-883e788bede6"). InnerVolumeSpecName "kube-api-access-l5wzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:14.399560 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.399498 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-metrics-certs" pod "e6ccd829-3732-4e8a-a8c5-883e788bede6" (UID: "e6ccd829-3732-4e8a-a8c5-883e788bede6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:26:14.399791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.399769 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls" pod "e6ccd829-3732-4e8a-a8c5-883e788bede6" (UID: "e6ccd829-3732-4e8a-a8c5-883e788bede6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:26:14.522951 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.522867 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5wzp\" (UniqueName: \"kubernetes.io/projected/e6ccd829-3732-4e8a-a8c5-883e788bede6-kube-api-access-l5wzp\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:14.522951 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.522901 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:14.522951 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.522916 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6ccd829-3732-4e8a-a8c5-883e788bede6-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:14.807553 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.807519 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn"] Oct 31 15:26:14.811260 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.811239 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-da9920-7bfdf6864b-4kgpn"] Oct 31 15:26:14.896273 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:14.896235 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ccd829-3732-4e8a-a8c5-883e788bede6" path="/var/lib/kubelet/pods/e6ccd829-3732-4e8a-a8c5-883e788bede6/volumes" Oct 31 15:26:21.511723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:21.510792 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:26:21.511723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:21.510881 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:21.674724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:21.673896 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:22.479730 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:22.470168 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:22.680958 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:22.680900 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:22.808808 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:22.794824 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:22.810284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:22.809140 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:26:23.230014 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:23.229983 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:26:26.261989 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:26:26.261956 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:26:43.230727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:43.194312 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:43.495991 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:43.495861 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:43.667324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:43.662800 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:44.235686 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:44.235641 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 15:26:44.236521 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:44.236495 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:26:44.270574 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:44.270547 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 15:26:45.114979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.114947 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n"] Oct 31 15:26:45.115253 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.115155 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" podUID="f8d8ca03-af52-438c-9c1a-53a28cd20ae8" containerName="maas-api" containerID="cri-o://8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b" gracePeriod=30 Oct 31 15:26:45.459489 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.459031 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:26:45.620076 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.620046 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-openshift-service-ca\") pod \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " Oct 31 15:26:45.620247 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.620090 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-metrics-certs\") pod \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " Oct 31 15:26:45.620247 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.620213 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-kube-api-access-knqvt\" (UniqueName: \"kubernetes.io/projected/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-kube-api-access-knqvt\") pod \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " Oct 31 15:26:45.620387 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.620332 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls\") pod \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\" (UID: \"f8d8ca03-af52-438c-9c1a-53a28cd20ae8\") " Oct 31 15:26:45.620486 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.620465 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-openshift-service-ca" pod "f8d8ca03-af52-438c-9c1a-53a28cd20ae8" (UID: "f8d8ca03-af52-438c-9c1a-53a28cd20ae8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:26:45.620822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.620798 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:45.622898 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.622870 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls" pod "f8d8ca03-af52-438c-9c1a-53a28cd20ae8" (UID: "f8d8ca03-af52-438c-9c1a-53a28cd20ae8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:26:45.623184 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.623142 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-kube-api-access-knqvt" pod "f8d8ca03-af52-438c-9c1a-53a28cd20ae8" (UID: "f8d8ca03-af52-438c-9c1a-53a28cd20ae8"). InnerVolumeSpecName "kube-api-access-knqvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:26:45.625187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.625109 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-metrics-certs" pod "f8d8ca03-af52-438c-9c1a-53a28cd20ae8" (UID: "f8d8ca03-af52-438c-9c1a-53a28cd20ae8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:26:45.739734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.738720 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knqvt\" (UniqueName: \"kubernetes.io/projected/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-kube-api-access-knqvt\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:45.739734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.738763 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:45.739734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:45.738779 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8d8ca03-af52-438c-9c1a-53a28cd20ae8-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:26:46.215879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.215234 4399 generic.go:356] "Generic (PLEG): container finished" podID="f8d8ca03-af52-438c-9c1a-53a28cd20ae8" containerID="8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b" exitCode=0 Oct 31 15:26:46.215879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.215378 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" event={"ID":"f8d8ca03-af52-438c-9c1a-53a28cd20ae8","Type":"ContainerDied","Data":"8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b"} Oct 31 15:26:46.215879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.215424 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" event={"ID":"f8d8ca03-af52-438c-9c1a-53a28cd20ae8","Type":"ContainerDied","Data":"f441ba32c73d713d795a30a72c828c063cee1ea7135aea6238c1fbc9e0d8b755"} Oct 31 15:26:46.215879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.215454 4399 scope.go:122] "RemoveContainer" containerID="8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b" Oct 31 15:26:46.215879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.215603 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n" Oct 31 15:26:46.266712 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.266676 4399 scope.go:122] "RemoveContainer" containerID="8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b" Oct 31 15:26:46.267199 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:26:46.267169 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b\": container with ID starting with 8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b not found: ID does not exist" containerID="8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b" Oct 31 15:26:46.267377 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.267358 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b"} err="failed to get container status \"8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b\": rpc error: code = NotFound desc = could not find container \"8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b\": container with ID starting with 8e41509ae0e13e18bd7d9fd8e001477e0182206b6eda3ab601fd2708c3724e0b not found: ID does not exist" Oct 31 15:26:46.386817 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.386780 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n"] Oct 31 15:26:46.437745 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.437714 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bd956c-6cf98fddd9-wbb7n"] Oct 31 15:26:46.898128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:46.898089 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d8ca03-af52-438c-9c1a-53a28cd20ae8" path="/var/lib/kubelet/pods/f8d8ca03-af52-438c-9c1a-53a28cd20ae8/volumes" Oct 31 15:26:56.207666 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:26:56.207628 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:26:56.952334 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.952307 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk"] Oct 31 15:26:56.952536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.952515 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" containerID="cri-o://78e36ab7853c6633aa83a4678a5f13f233fbf24a5d08f2c0d442c7fb33892e46" gracePeriod=30 Oct 31 15:26:56.955092 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.955070 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548"] Oct 31 15:26:56.955718 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.955681 4399 cpu_manager.go:414] "containerMap: removing container" podUID="f8d8ca03-af52-438c-9c1a-53a28cd20ae8" containerName="maas-api" Oct 31 15:26:56.955718 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.955715 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d8ca03-af52-438c-9c1a-53a28cd20ae8" containerName="maas-api" Oct 31 15:26:56.955890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.955760 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e6ccd829-3732-4e8a-a8c5-883e788bede6" containerName="maas-api" Oct 31 15:26:56.955890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.955765 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ccd829-3732-4e8a-a8c5-883e788bede6" containerName="maas-api" Oct 31 15:26:56.955890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.955860 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d8ca03-af52-438c-9c1a-53a28cd20ae8" containerName="maas-api" Oct 31 15:26:56.955890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.955879 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ccd829-3732-4e8a-a8c5-883e788bede6" containerName="maas-api" Oct 31 15:26:56.959180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:56.959160 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.027824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.027797 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548"] Oct 31 15:26:57.035666 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.035638 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.035850 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.035684 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.035850 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.035732 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.035971 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.035925 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.036059 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.036041 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.036153 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.036139 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.036211 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.036175 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.036267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.036216 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tlr\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-kube-api-access-65tlr\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.036325 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.036306 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.048844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.048824 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d9cddd-openshift-default-dockercfg-7h672\"" Oct 31 15:26:57.141731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.137659 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.141731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.137801 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.141731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.137894 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.141731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.137927 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.141731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.137964 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tlr\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-kube-api-access-65tlr\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.141731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.138034 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.141731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.138066 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.141731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.138098 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.141731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.138128 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.143644 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.143606 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.144123 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.144076 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.144880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.144845 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.145737 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.145686 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.146102 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.146078 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.163061 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.148253 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.163061 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.148957 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.211847 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.211526 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tlr\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-kube-api-access-65tlr\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.218729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.215903 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.273676 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.273646 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:57.341367 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.338437 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:26:57.341367 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.338511 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:26:57.706790 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.706757 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548"] Oct 31 15:26:57.708347 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:26:57.708316 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ceb65ff_52fb_40c2_ac9b_cf37657571e2.slice/crio-56843b20a77294b6d88a905f5479b0aba44526bb2315069c8f3294305d32215c WatchSource:0}: Error finding container 56843b20a77294b6d88a905f5479b0aba44526bb2315069c8f3294305d32215c: Status 404 returned error can't find the container with id 56843b20a77294b6d88a905f5479b0aba44526bb2315069c8f3294305d32215c Oct 31 15:26:57.710740 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.710670 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:26:57.710838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.710774 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:26:57.710838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:57.710805 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:26:58.350581 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:58.350534 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" event={"ID":"5ceb65ff-52fb-40c2-ac9b-cf37657571e2","Type":"ContainerStarted","Data":"19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5"} Oct 31 15:26:58.351268 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:58.351225 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" event={"ID":"5ceb65ff-52fb-40c2-ac9b-cf37657571e2","Type":"ContainerStarted","Data":"56843b20a77294b6d88a905f5479b0aba44526bb2315069c8f3294305d32215c"} Oct 31 15:26:58.473710 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:58.473630 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" podStartSLOduration=2.473615537 podStartE2EDuration="2.473615537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:26:58.471296359 +0000 UTC m=+2461.682537772" watchObservedRunningTime="2026-10-31 15:26:58.473615537 +0000 UTC m=+2461.684856948" Oct 31 15:26:59.274741 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:59.274683 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:26:59.276114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:59.276092 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 15:26:59.276169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:26:59.276138 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 15:27:00.274246 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:00.274210 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 15:27:00.274753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:00.274260 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 15:27:01.274482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:01.274447 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 15:27:01.274951 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:01.274507 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 15:27:02.285315 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.284311 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 15:27:02.285315 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.284368 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 15:27:02.397773 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.397737 4399 generic.go:356] "Generic (PLEG): container finished" podID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerID="78e36ab7853c6633aa83a4678a5f13f233fbf24a5d08f2c0d442c7fb33892e46" exitCode=0 Oct 31 15:27:02.397773 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.397757 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" event={"ID":"e07de2d1-ed86-42d3-9da3-2f38a7eea553","Type":"ContainerDied","Data":"78e36ab7853c6633aa83a4678a5f13f233fbf24a5d08f2c0d442c7fb33892e46"} Oct 31 15:27:02.397983 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.397823 4399 scope.go:122] "RemoveContainer" containerID="95da7c94907847e5792c2cd328016380cfb5259377614fe59cf01b23e7f658d1" Oct 31 15:27:02.475451 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.475422 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:27:02.622060 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.621978 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-credential-socket\") pod \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " Oct 31 15:27:02.622060 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622039 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-token\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-token\") pod \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " Oct 31 15:27:02.622273 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622114 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istiod-ca-cert\") pod \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " Oct 31 15:27:02.622273 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622163 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-certs\") pod \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " Oct 31 15:27:02.622273 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622222 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-socket\") pod \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " Oct 31 15:27:02.622273 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622266 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-kube-api-access-2trrk\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-kube-api-access-2trrk\") pod \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " Oct 31 15:27:02.622489 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622322 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-envoy\") pod \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " Oct 31 15:27:02.622489 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622332 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-credential-socket" pod "e07de2d1-ed86-42d3-9da3-2f38a7eea553" (UID: "e07de2d1-ed86-42d3-9da3-2f38a7eea553"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:02.622489 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622376 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-podinfo\") pod \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " Oct 31 15:27:02.622489 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622402 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-data\") pod \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\" (UID: \"e07de2d1-ed86-42d3-9da3-2f38a7eea553\") " Oct 31 15:27:02.622680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622546 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-certs" pod "e07de2d1-ed86-42d3-9da3-2f38a7eea553" (UID: "e07de2d1-ed86-42d3-9da3-2f38a7eea553"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:02.622680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622564 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-socket" pod "e07de2d1-ed86-42d3-9da3-2f38a7eea553" (UID: "e07de2d1-ed86-42d3-9da3-2f38a7eea553"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:02.622680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622601 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istiod-ca-cert" pod "e07de2d1-ed86-42d3-9da3-2f38a7eea553" (UID: "e07de2d1-ed86-42d3-9da3-2f38a7eea553"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:27:02.622878 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.622806 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-data" pod "e07de2d1-ed86-42d3-9da3-2f38a7eea553" (UID: "e07de2d1-ed86-42d3-9da3-2f38a7eea553"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:02.623031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.623012 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:02.623031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.623036 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:02.623031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.623051 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:02.623031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.623066 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:02.623432 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.623081 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:02.624933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.624902 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-podinfo" pod "e07de2d1-ed86-42d3-9da3-2f38a7eea553" (UID: "e07de2d1-ed86-42d3-9da3-2f38a7eea553"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:27:02.624933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.624911 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-envoy" pod "e07de2d1-ed86-42d3-9da3-2f38a7eea553" (UID: "e07de2d1-ed86-42d3-9da3-2f38a7eea553"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:27:02.625080 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.624944 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-token" pod "e07de2d1-ed86-42d3-9da3-2f38a7eea553" (UID: "e07de2d1-ed86-42d3-9da3-2f38a7eea553"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:02.625080 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.624958 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-kube-api-access-2trrk" pod "e07de2d1-ed86-42d3-9da3-2f38a7eea553" (UID: "e07de2d1-ed86-42d3-9da3-2f38a7eea553"). InnerVolumeSpecName "kube-api-access-2trrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:02.724378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.724344 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:02.724378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.724374 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:02.724378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.724383 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2trrk\" (UniqueName: \"kubernetes.io/projected/e07de2d1-ed86-42d3-9da3-2f38a7eea553-kube-api-access-2trrk\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:02.724619 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:02.724395 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e07de2d1-ed86-42d3-9da3-2f38a7eea553-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:03.274143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:03.274112 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body= Oct 31 15:27:03.274361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:03.274169 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" Oct 31 15:27:03.404933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:03.404903 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" event={"ID":"e07de2d1-ed86-42d3-9da3-2f38a7eea553","Type":"ContainerDied","Data":"ad65091adb24548f23e4a79f3d065be8c781b8f3bd0fe81fe675011075e1bf64"} Oct 31 15:27:03.405395 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:03.404936 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk" Oct 31 15:27:03.405395 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:03.404960 4399 scope.go:122] "RemoveContainer" containerID="78e36ab7853c6633aa83a4678a5f13f233fbf24a5d08f2c0d442c7fb33892e46" Oct 31 15:27:03.576689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:03.576604 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk"] Oct 31 15:27:03.582090 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:03.582058 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bd956c-openshift-default-864b477997-2tkhk"] Oct 31 15:27:04.459493 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:04.459462 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:27:04.460092 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:04.459822 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:27:04.461081 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:04.461056 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:27:04.897500 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:04.897424 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" path="/var/lib/kubelet/pods/e07de2d1-ed86-42d3-9da3-2f38a7eea553/volumes" Oct 31 15:27:13.668208 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:13.666790 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:27:14.169196 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:14.169147 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 15:27:22.329013 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.328979 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9"] Oct 31 15:27:22.329546 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.329179 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" podUID="f3124d69-2c22-44e9-9a87-17a793270206" containerName="maas-api" containerID="cri-o://07f71bab10bacecff3ada076d1c242edbc4c6cfa63177850126df58aa9562dc5" gracePeriod=30 Oct 31 15:27:22.652202 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.652107 4399 generic.go:356] "Generic (PLEG): container finished" podID="f3124d69-2c22-44e9-9a87-17a793270206" containerID="07f71bab10bacecff3ada076d1c242edbc4c6cfa63177850126df58aa9562dc5" exitCode=0 Oct 31 15:27:22.652511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.652478 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" event={"ID":"f3124d69-2c22-44e9-9a87-17a793270206","Type":"ContainerDied","Data":"07f71bab10bacecff3ada076d1c242edbc4c6cfa63177850126df58aa9562dc5"} Oct 31 15:27:22.683019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.682992 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:27:22.730199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.730158 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3124d69-2c22-44e9-9a87-17a793270206-kube-api-access-dtf8r\" (UniqueName: \"kubernetes.io/projected/f3124d69-2c22-44e9-9a87-17a793270206-kube-api-access-dtf8r\") pod \"f3124d69-2c22-44e9-9a87-17a793270206\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " Oct 31 15:27:22.736585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.736552 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3124d69-2c22-44e9-9a87-17a793270206-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3124d69-2c22-44e9-9a87-17a793270206-openshift-service-ca\") pod \"f3124d69-2c22-44e9-9a87-17a793270206\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " Oct 31 15:27:22.744474 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.744428 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-maas-api-tls\") pod \"f3124d69-2c22-44e9-9a87-17a793270206\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " Oct 31 15:27:22.752040 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.751996 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-metrics-certs\") pod \"f3124d69-2c22-44e9-9a87-17a793270206\" (UID: \"f3124d69-2c22-44e9-9a87-17a793270206\") " Oct 31 15:27:22.758894 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.738248 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3124d69-2c22-44e9-9a87-17a793270206-openshift-service-ca" pod "f3124d69-2c22-44e9-9a87-17a793270206" (UID: "f3124d69-2c22-44e9-9a87-17a793270206"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:27:22.764477 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.759864 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3124d69-2c22-44e9-9a87-17a793270206-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:22.764691 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.762923 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-maas-api-tls" pod "f3124d69-2c22-44e9-9a87-17a793270206" (UID: "f3124d69-2c22-44e9-9a87-17a793270206"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:27:22.768900 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.767281 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-metrics-certs" pod "f3124d69-2c22-44e9-9a87-17a793270206" (UID: "f3124d69-2c22-44e9-9a87-17a793270206"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:27:22.773048 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.773016 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3124d69-2c22-44e9-9a87-17a793270206-kube-api-access-dtf8r" pod "f3124d69-2c22-44e9-9a87-17a793270206" (UID: "f3124d69-2c22-44e9-9a87-17a793270206"). InnerVolumeSpecName "kube-api-access-dtf8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:27:22.866072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.866036 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:22.866072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.866072 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f3124d69-2c22-44e9-9a87-17a793270206-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:22.866315 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:22.866087 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtf8r\" (UniqueName: \"kubernetes.io/projected/f3124d69-2c22-44e9-9a87-17a793270206-kube-api-access-dtf8r\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:27:23.659740 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:23.659689 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" Oct 31 15:27:23.659740 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:23.659723 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9" event={"ID":"f3124d69-2c22-44e9-9a87-17a793270206","Type":"ContainerDied","Data":"afecf4cee02d9fa74e78925a2739a917f58536606fb11a4051b8b48302ffa94e"} Oct 31 15:27:23.660220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:23.659781 4399 scope.go:122] "RemoveContainer" containerID="07f71bab10bacecff3ada076d1c242edbc4c6cfa63177850126df58aa9562dc5" Oct 31 15:27:23.823415 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:23.823385 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9"] Oct 31 15:27:23.872920 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:23.872890 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c4ede8-7fcb66f77c-582t9"] Oct 31 15:27:24.896398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:24.896366 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3124d69-2c22-44e9-9a87-17a793270206" path="/var/lib/kubelet/pods/f3124d69-2c22-44e9-9a87-17a793270206/volumes" Oct 31 15:27:26.206939 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:27:26.206906 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:27:28.707481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.707446 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p"] Oct 31 15:27:28.708026 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.708013 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" Oct 31 15:27:28.708079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.708028 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" Oct 31 15:27:28.708079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.708053 4399 cpu_manager.go:414] "containerMap: removing container" podUID="f3124d69-2c22-44e9-9a87-17a793270206" containerName="maas-api" Oct 31 15:27:28.708079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.708057 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3124d69-2c22-44e9-9a87-17a793270206" containerName="maas-api" Oct 31 15:27:28.708079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.708066 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" Oct 31 15:27:28.708079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.708072 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" Oct 31 15:27:28.708238 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.708164 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" Oct 31 15:27:28.708238 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.708175 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3124d69-2c22-44e9-9a87-17a793270206" containerName="maas-api" Oct 31 15:27:28.708238 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.708181 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07de2d1-ed86-42d3-9da3-2f38a7eea553" containerName="istio-proxy" Oct 31 15:27:28.713018 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.713001 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.767233 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.767206 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p"] Oct 31 15:27:28.788599 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.788572 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfk9\" (UniqueName: \"kubernetes.io/projected/2f56fdda-d597-4e66-9e41-77092b8f99d9-kube-api-access-bjfk9\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.788814 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.788604 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f56fdda-d597-4e66-9e41-77092b8f99d9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.788814 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.788783 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.788899 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.788893 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.813128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.813094 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-531c3c-serving-cert\"" Oct 31 15:27:28.890015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.889986 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.890166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.890109 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.890166 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:27:28.890125 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-serving-cert: secret "maas-api-e2e-shared-a-w3-531c3c-serving-cert" not found Oct 31 15:27:28.890255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.890182 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfk9\" (UniqueName: \"kubernetes.io/projected/2f56fdda-d597-4e66-9e41-77092b8f99d9-kube-api-access-bjfk9\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.890255 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:27:28.890194 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls podName:2f56fdda-d597-4e66-9e41-77092b8f99d9 nodeName:}" failed. No retries permitted until 2026-10-31 15:27:29.390177721 +0000 UTC m=+2492.601419112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls") pod "maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" (UID: "2f56fdda-d597-4e66-9e41-77092b8f99d9") : secret "maas-api-e2e-shared-a-w3-531c3c-serving-cert" not found Oct 31 15:27:28.890255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.890222 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f56fdda-d597-4e66-9e41-77092b8f99d9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.890972 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.890884 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f56fdda-d597-4e66-9e41-77092b8f99d9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.892735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.892710 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:28.947035 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:28.947007 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfk9\" (UniqueName: \"kubernetes.io/projected/2f56fdda-d597-4e66-9e41-77092b8f99d9-kube-api-access-bjfk9\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:29.397967 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:29.397934 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:29.400521 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:29.400494 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:29.626958 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:29.626924 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:30.002822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:30.002783 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p"] Oct 31 15:27:30.006258 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:27:30.006227 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f56fdda_d597_4e66_9e41_77092b8f99d9.slice/crio-73d5312a2e4c0f84f3f49cec55b6226e1efacef4f202d6d343b8dbe5e9af26f0 WatchSource:0}: Error finding container 73d5312a2e4c0f84f3f49cec55b6226e1efacef4f202d6d343b8dbe5e9af26f0: Status 404 returned error can't find the container with id 73d5312a2e4c0f84f3f49cec55b6226e1efacef4f202d6d343b8dbe5e9af26f0 Oct 31 15:27:30.708423 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:30.708381 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" event={"ID":"2f56fdda-d597-4e66-9e41-77092b8f99d9","Type":"ContainerStarted","Data":"73d5312a2e4c0f84f3f49cec55b6226e1efacef4f202d6d343b8dbe5e9af26f0"} Oct 31 15:27:31.717245 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:31.717209 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" event={"ID":"2f56fdda-d597-4e66-9e41-77092b8f99d9","Type":"ContainerStarted","Data":"a17d5bae9d27ec4b044641396c99fee0b41b6b00656c8914d51d02814d06e85b"} Oct 31 15:27:31.717645 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:31.717279 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:31.851643 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:31.851594 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" podStartSLOduration=2.847962154 podStartE2EDuration="3.851579717s" totalImagesPullingTime="1.003617563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:31.84754057 +0000 UTC m=+2495.058781982" watchObservedRunningTime="2026-10-31 15:27:31.851579717 +0000 UTC m=+2495.062821128" Oct 31 15:27:37.729356 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:37.729327 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 15:27:39.082851 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.082822 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q"] Oct 31 15:27:39.086995 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.086978 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.143302 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.143275 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q"] Oct 31 15:27:39.188455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.188429 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-e72453-serving-cert\"" Oct 31 15:27:39.199541 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.199506 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.199671 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.199552 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjzns\" (UniqueName: \"kubernetes.io/projected/3241e52d-5b4e-40eb-b28d-e88c40468457-kube-api-access-kjzns\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.199859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.199837 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3241e52d-5b4e-40eb-b28d-e88c40468457-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.200035 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.199897 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.301188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.301152 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.301357 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.301204 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjzns\" (UniqueName: \"kubernetes.io/projected/3241e52d-5b4e-40eb-b28d-e88c40468457-kube-api-access-kjzns\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.301357 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.301327 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3241e52d-5b4e-40eb-b28d-e88c40468457-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.301484 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.301371 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.301549 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:27:39.301485 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-serving-cert: secret "maas-api-e2e-shared-a-w1-e72453-serving-cert" not found Oct 31 15:27:39.301608 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:27:39.301563 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls podName:3241e52d-5b4e-40eb-b28d-e88c40468457 nodeName:}" failed. No retries permitted until 2026-10-31 15:27:39.801543247 +0000 UTC m=+2503.012784640 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls") pod "maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" (UID: "3241e52d-5b4e-40eb-b28d-e88c40468457") : secret "maas-api-e2e-shared-a-w1-e72453-serving-cert" not found Oct 31 15:27:39.302017 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.301993 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3241e52d-5b4e-40eb-b28d-e88c40468457-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.303816 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.303798 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.360612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.360539 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjzns\" (UniqueName: \"kubernetes.io/projected/3241e52d-5b4e-40eb-b28d-e88c40468457-kube-api-access-kjzns\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.814469 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.813521 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:39.839326 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:39.839293 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:40.000272 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:40.000238 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:40.451266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:40.451239 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q"] Oct 31 15:27:40.453795 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:27:40.453769 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3241e52d_5b4e_40eb_b28d_e88c40468457.slice/crio-2a8ea18bc47ae9056198cab7b3ef5794fc23a4f3da0e0c56ba86dbd949513d59 WatchSource:0}: Error finding container 2a8ea18bc47ae9056198cab7b3ef5794fc23a4f3da0e0c56ba86dbd949513d59: Status 404 returned error can't find the container with id 2a8ea18bc47ae9056198cab7b3ef5794fc23a4f3da0e0c56ba86dbd949513d59 Oct 31 15:27:40.775040 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:40.774955 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" event={"ID":"3241e52d-5b4e-40eb-b28d-e88c40468457","Type":"ContainerStarted","Data":"2a8ea18bc47ae9056198cab7b3ef5794fc23a4f3da0e0c56ba86dbd949513d59"} Oct 31 15:27:41.783201 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:41.783160 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" event={"ID":"3241e52d-5b4e-40eb-b28d-e88c40468457","Type":"ContainerStarted","Data":"38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592"} Oct 31 15:27:41.783646 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:41.783324 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:41.899472 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:41.899422 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" podStartSLOduration=1.998963754 podStartE2EDuration="2.899408388s" totalImagesPullingTime="900.444634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:41.896892406 +0000 UTC m=+2505.108133817" watchObservedRunningTime="2026-10-31 15:27:41.899408388 +0000 UTC m=+2505.110649800" Oct 31 15:27:47.795441 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:47.795411 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 15:27:49.267574 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.267536 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8"] Oct 31 15:27:49.271544 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.271528 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.374104 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.374062 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-21111f-serving-cert\"" Oct 31 15:27:49.377320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.377298 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8"] Oct 31 15:27:49.464231 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.464198 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wzg5\" (UniqueName: \"kubernetes.io/projected/519232b6-f149-4927-ad3d-7d2224fd47f9-kube-api-access-2wzg5\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.464388 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.464257 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.464388 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.464354 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.464494 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.464474 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519232b6-f149-4927-ad3d-7d2224fd47f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.568532 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.568438 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519232b6-f149-4927-ad3d-7d2224fd47f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.570358 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.570329 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wzg5\" (UniqueName: \"kubernetes.io/projected/519232b6-f149-4927-ad3d-7d2224fd47f9-kube-api-access-2wzg5\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.570947 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.570925 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.571138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.571123 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.571776 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.569463 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519232b6-f149-4927-ad3d-7d2224fd47f9-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.571996 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:27:49.571980 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-serving-cert: secret "maas-api-e2e-shared-a-w4-21111f-serving-cert" not found Oct 31 15:27:49.572155 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:27:49.572144 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls podName:519232b6-f149-4927-ad3d-7d2224fd47f9 nodeName:}" failed. No retries permitted until 2026-10-31 15:27:50.072121389 +0000 UTC m=+2513.283362786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls") pod "maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" (UID: "519232b6-f149-4927-ad3d-7d2224fd47f9") : secret "maas-api-e2e-shared-a-w4-21111f-serving-cert" not found Oct 31 15:27:49.578970 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.578947 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:49.627145 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:49.627119 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wzg5\" (UniqueName: \"kubernetes.io/projected/519232b6-f149-4927-ad3d-7d2224fd47f9-kube-api-access-2wzg5\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:50.087850 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:50.087812 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:50.090604 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:50.090572 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:50.185177 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:50.185152 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:50.562457 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:50.562431 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8"] Oct 31 15:27:50.564969 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:27:50.564945 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod519232b6_f149_4927_ad3d_7d2224fd47f9.slice/crio-27c8d211efbe00f7a70993933de8fab4377790c32e74e2ea8fa71a220ef1fe2a WatchSource:0}: Error finding container 27c8d211efbe00f7a70993933de8fab4377790c32e74e2ea8fa71a220ef1fe2a: Status 404 returned error can't find the container with id 27c8d211efbe00f7a70993933de8fab4377790c32e74e2ea8fa71a220ef1fe2a Oct 31 15:27:50.836422 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:50.836334 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" event={"ID":"519232b6-f149-4927-ad3d-7d2224fd47f9","Type":"ContainerStarted","Data":"27c8d211efbe00f7a70993933de8fab4377790c32e74e2ea8fa71a220ef1fe2a"} Oct 31 15:27:51.843566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:51.843531 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" event={"ID":"519232b6-f149-4927-ad3d-7d2224fd47f9","Type":"ContainerStarted","Data":"ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799"} Oct 31 15:27:51.844025 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:51.843622 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:51.958622 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:51.958574 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" podStartSLOduration=2.128085267 podStartE2EDuration="2.958559936s" totalImagesPullingTime="830.474669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:51.957444318 +0000 UTC m=+2515.168685733" watchObservedRunningTime="2026-10-31 15:27:51.958559936 +0000 UTC m=+2515.169801348" Oct 31 15:27:56.207189 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:27:56.207158 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:27:57.400941 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.400903 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj"] Oct 31 15:27:57.405234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.405215 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.499396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.499366 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.499607 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.499436 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.499607 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.499538 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080b36f-b381-43a4-a553-a7de44d4961f-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.499607 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.499590 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.499825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.499692 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080b36f-b381-43a4-a553-a7de44d4961f-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.499825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.499804 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.499931 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.499866 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.499990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.499931 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb8q8\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-kube-api-access-vb8q8\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.500043 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.499998 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.510514 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.510487 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj"] Oct 31 15:27:57.600797 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.600767 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.600969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.600812 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080b36f-b381-43a4-a553-a7de44d4961f-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.600969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.600877 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.600969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.600924 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.600983 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb8q8\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-kube-api-access-vb8q8\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.601046 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.601085 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.601148 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.601193 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080b36f-b381-43a4-a553-a7de44d4961f-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.601231 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-credential-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601469 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.601406 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-data\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601469 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.601427 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-certs\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601622 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.601598 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-socket\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.601683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.601657 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080b36f-b381-43a4-a553-a7de44d4961f-istiod-ca-cert\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.603469 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.603443 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-envoy\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.603685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.603669 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080b36f-b381-43a4-a553-a7de44d4961f-istio-podinfo\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.657266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.657205 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-istio-token\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.657365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.657267 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb8q8\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-kube-api-access-vb8q8\") pod \"e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.718882 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.718860 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:57.855451 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:57.855417 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:27:58.148710 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:58.148667 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj"] Oct 31 15:27:58.149615 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:27:58.149590 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7080b36f_b381_43a4_a553_a7de44d4961f.slice/crio-c3f35a7982df8bb4bd018642b0259f6c9df43c8f0b08a8b39c64d79ff597f79c WatchSource:0}: Error finding container c3f35a7982df8bb4bd018642b0259f6c9df43c8f0b08a8b39c64d79ff597f79c: Status 404 returned error can't find the container with id c3f35a7982df8bb4bd018642b0259f6c9df43c8f0b08a8b39c64d79ff597f79c Oct 31 15:27:58.152207 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:58.152175 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:27:58.152305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:58.152239 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:27:58.152305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:58.152290 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:27:58.902057 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:58.901372 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" event={"ID":"7080b36f-b381-43a4-a553-a7de44d4961f","Type":"ContainerStarted","Data":"ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b"} Oct 31 15:27:58.902057 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:58.901412 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" event={"ID":"7080b36f-b381-43a4-a553-a7de44d4961f","Type":"ContainerStarted","Data":"c3f35a7982df8bb4bd018642b0259f6c9df43c8f0b08a8b39c64d79ff597f79c"} Oct 31 15:27:59.029981 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:59.029921 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" podStartSLOduration=2.029902531 podStartE2EDuration="2.029902531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:27:59.027830513 +0000 UTC m=+2522.239071926" watchObservedRunningTime="2026-10-31 15:27:59.029902531 +0000 UTC m=+2522.241143943" Oct 31 15:27:59.719807 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:59.719769 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:27:59.721141 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:59.721116 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 15:27:59.721270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:27:59.721167 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 15:28:00.719944 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:00.719906 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 15:28:00.720349 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:00.719975 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 15:28:01.719292 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:01.719256 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 15:28:01.719476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:01.719318 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 15:28:02.720144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:02.720107 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 15:28:02.720635 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:02.720168 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 15:28:03.720024 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:03.719979 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Oct 31 15:28:03.720216 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:03.720055 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Oct 31 15:28:04.723990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:04.723962 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:28:04.724383 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:04.724362 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:28:04.725656 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:28:04.725638 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:28:26.207350 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:28:26.207310 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:28:56.207693 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:28:56.207654 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:29:17.424180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.424149 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4"] Oct 31 15:29:17.424806 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.424377 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" podUID="d1c38736-f7c8-47b2-bb7d-3cff37743d02" containerName="maas-api" containerID="cri-o://9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806" gracePeriod=30 Oct 31 15:29:17.758285 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.758261 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:29:17.898185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.898146 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls\") pod \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " Oct 31 15:29:17.898834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.898808 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-metrics-certs\") pod \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " Oct 31 15:29:17.901288 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.899398 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c38736-f7c8-47b2-bb7d-3cff37743d02-kube-api-access-l954f\" (UniqueName: \"kubernetes.io/projected/d1c38736-f7c8-47b2-bb7d-3cff37743d02-kube-api-access-l954f\") pod \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " Oct 31 15:29:17.901594 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.901552 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c38736-f7c8-47b2-bb7d-3cff37743d02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c38736-f7c8-47b2-bb7d-3cff37743d02-openshift-service-ca\") pod \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\" (UID: \"d1c38736-f7c8-47b2-bb7d-3cff37743d02\") " Oct 31 15:29:17.902519 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.902496 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c38736-f7c8-47b2-bb7d-3cff37743d02-openshift-service-ca" pod "d1c38736-f7c8-47b2-bb7d-3cff37743d02" (UID: "d1c38736-f7c8-47b2-bb7d-3cff37743d02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:29:17.902626 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.902565 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-metrics-certs" pod "d1c38736-f7c8-47b2-bb7d-3cff37743d02" (UID: "d1c38736-f7c8-47b2-bb7d-3cff37743d02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:29:17.903601 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.903568 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c38736-f7c8-47b2-bb7d-3cff37743d02-kube-api-access-l954f" pod "d1c38736-f7c8-47b2-bb7d-3cff37743d02" (UID: "d1c38736-f7c8-47b2-bb7d-3cff37743d02"). InnerVolumeSpecName "kube-api-access-l954f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:17.904319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:17.904298 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls" pod "d1c38736-f7c8-47b2-bb7d-3cff37743d02" (UID: "d1c38736-f7c8-47b2-bb7d-3cff37743d02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:29:18.018444 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.018365 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c38736-f7c8-47b2-bb7d-3cff37743d02-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:18.018658 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.018642 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:18.018770 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.018758 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1c38736-f7c8-47b2-bb7d-3cff37743d02-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:18.018859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.018848 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l954f\" (UniqueName: \"kubernetes.io/projected/d1c38736-f7c8-47b2-bb7d-3cff37743d02-kube-api-access-l954f\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:18.433679 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.433604 4399 generic.go:356] "Generic (PLEG): container finished" podID="d1c38736-f7c8-47b2-bb7d-3cff37743d02" containerID="9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806" exitCode=0 Oct 31 15:29:18.435288 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.434408 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" event={"ID":"d1c38736-f7c8-47b2-bb7d-3cff37743d02","Type":"ContainerDied","Data":"9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806"} Oct 31 15:29:18.435501 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.435477 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" event={"ID":"d1c38736-f7c8-47b2-bb7d-3cff37743d02","Type":"ContainerDied","Data":"fda5e04792f898e350bf165fcffd0a00502b2e8f7df6cb7f5416a4c6ffb1ec6e"} Oct 31 15:29:18.435862 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.435844 4399 scope.go:122] "RemoveContainer" containerID="9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806" Oct 31 15:29:18.436162 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.434756 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4" Oct 31 15:29:18.476122 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.474587 4399 scope.go:122] "RemoveContainer" containerID="9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806" Oct 31 15:29:18.476539 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:18.476501 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806\": container with ID starting with 9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806 not found: ID does not exist" containerID="9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806" Oct 31 15:29:18.476682 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.476546 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806"} err="failed to get container status \"9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806\": rpc error: code = NotFound desc = could not find container \"9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806\": container with ID starting with 9a9a19e96bd6787993e967c0e9bfbea2ac6d0a443f8d3b9a50f8489ceff38806 not found: ID does not exist" Oct 31 15:29:18.606187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.606159 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4"] Oct 31 15:29:18.655644 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.655615 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-d9cddd-867b4d684f-slgh4"] Oct 31 15:29:18.897759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:18.897723 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c38736-f7c8-47b2-bb7d-3cff37743d02" path="/var/lib/kubelet/pods/d1c38736-f7c8-47b2-bb7d-3cff37743d02/volumes" Oct 31 15:29:19.648954 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:19.648924 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj"] Oct 31 15:29:19.649342 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:19.649125 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" containerID="cri-o://ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b" gracePeriod=30 Oct 31 15:29:19.656082 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:19.656049 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548"] Oct 31 15:29:19.656267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:19.656227 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerName="istio-proxy" containerID="cri-o://19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5" gracePeriod=30 Oct 31 15:29:19.726798 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:19.726761 4399 patch_prober.go:29] interesting pod/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:29:19.726953 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:19.726827 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:29:24.201060 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.201027 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn"] Oct 31 15:29:24.201633 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.201619 4399 cpu_manager.go:414] "containerMap: removing container" podUID="d1c38736-f7c8-47b2-bb7d-3cff37743d02" containerName="maas-api" Oct 31 15:29:24.201633 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.201634 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c38736-f7c8-47b2-bb7d-3cff37743d02" containerName="maas-api" Oct 31 15:29:24.201774 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.201749 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c38736-f7c8-47b2-bb7d-3cff37743d02" containerName="maas-api" Oct 31 15:29:24.205206 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.205192 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.293483 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.293449 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.293631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.293553 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq862\" (UniqueName: \"kubernetes.io/projected/e65e4c6b-bd54-420c-b113-fb3ead82543b-kube-api-access-vq862\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.293631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.293575 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.293830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.293730 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65e4c6b-bd54-420c-b113-fb3ead82543b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.307114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.307089 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-55b3fb-serving-cert\"" Oct 31 15:29:24.310432 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.310413 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn"] Oct 31 15:29:24.395450 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.395417 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq862\" (UniqueName: \"kubernetes.io/projected/e65e4c6b-bd54-420c-b113-fb3ead82543b-kube-api-access-vq862\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.395663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.395460 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.395663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.395620 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65e4c6b-bd54-420c-b113-fb3ead82543b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.395837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.395749 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.395881 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:24.395866 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-serving-cert: secret "maas-api-e2e-shared-b-w3-55b3fb-serving-cert" not found Oct 31 15:29:24.395938 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:24.395927 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls podName:e65e4c6b-bd54-420c-b113-fb3ead82543b nodeName:}" failed. No retries permitted until 2026-10-31 15:29:24.89591111 +0000 UTC m=+2608.107152499 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls") pod "maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" (UID: "e65e4c6b-bd54-420c-b113-fb3ead82543b") : secret "maas-api-e2e-shared-b-w3-55b3fb-serving-cert" not found Oct 31 15:29:24.396294 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.396276 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65e4c6b-bd54-420c-b113-fb3ead82543b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.398119 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.398100 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.485606 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.485530 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq862\" (UniqueName: \"kubernetes.io/projected/e65e4c6b-bd54-420c-b113-fb3ead82543b-kube-api-access-vq862\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.905405 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.905374 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.908367 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.908343 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:24.918194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.918175 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:29:24.951225 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:24.951207 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:29:25.006436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006413 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-credential-socket\") pod \"7080b36f-b381-43a4-a553-a7de44d4961f\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " Oct 31 15:29:25.006572 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006464 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-socket\") pod \"7080b36f-b381-43a4-a553-a7de44d4961f\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " Oct 31 15:29:25.006572 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006496 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7080b36f-b381-43a4-a553-a7de44d4961f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080b36f-b381-43a4-a553-a7de44d4961f-istio-podinfo\") pod \"7080b36f-b381-43a4-a553-a7de44d4961f\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " Oct 31 15:29:25.006572 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006515 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7080b36f-b381-43a4-a553-a7de44d4961f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080b36f-b381-43a4-a553-a7de44d4961f-istiod-ca-cert\") pod \"7080b36f-b381-43a4-a553-a7de44d4961f\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " Oct 31 15:29:25.006791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006633 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-istio-token\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-istio-token\") pod \"7080b36f-b381-43a4-a553-a7de44d4961f\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " Oct 31 15:29:25.006791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006656 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-kube-api-access-vb8q8\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-kube-api-access-vb8q8\") pod \"7080b36f-b381-43a4-a553-a7de44d4961f\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " Oct 31 15:29:25.006791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006689 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-credential-socket" pod "7080b36f-b381-43a4-a553-a7de44d4961f" (UID: "7080b36f-b381-43a4-a553-a7de44d4961f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.006791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006739 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-envoy\") pod \"7080b36f-b381-43a4-a553-a7de44d4961f\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " Oct 31 15:29:25.006791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006770 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-certs\") pod \"7080b36f-b381-43a4-a553-a7de44d4961f\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " Oct 31 15:29:25.007054 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006798 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-socket" pod "7080b36f-b381-43a4-a553-a7de44d4961f" (UID: "7080b36f-b381-43a4-a553-a7de44d4961f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.007054 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006873 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-data\") pod \"7080b36f-b381-43a4-a553-a7de44d4961f\" (UID: \"7080b36f-b381-43a4-a553-a7de44d4961f\") " Oct 31 15:29:25.007054 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.006982 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7080b36f-b381-43a4-a553-a7de44d4961f-istiod-ca-cert" pod "7080b36f-b381-43a4-a553-a7de44d4961f" (UID: "7080b36f-b381-43a4-a553-a7de44d4961f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:29:25.007844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.007338 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-data" pod "7080b36f-b381-43a4-a553-a7de44d4961f" (UID: "7080b36f-b381-43a4-a553-a7de44d4961f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.007844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.007292 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-certs" pod "7080b36f-b381-43a4-a553-a7de44d4961f" (UID: "7080b36f-b381-43a4-a553-a7de44d4961f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.007844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.007684 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.007844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.007721 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.007844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.007736 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.007844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.007752 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.007844 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.007768 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7080b36f-b381-43a4-a553-a7de44d4961f-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.009073 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.009024 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-kube-api-access-vb8q8" pod "7080b36f-b381-43a4-a553-a7de44d4961f" (UID: "7080b36f-b381-43a4-a553-a7de44d4961f"). InnerVolumeSpecName "kube-api-access-vb8q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:25.009399 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.009382 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-envoy" pod "7080b36f-b381-43a4-a553-a7de44d4961f" (UID: "7080b36f-b381-43a4-a553-a7de44d4961f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.009581 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.009554 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-istio-token" pod "7080b36f-b381-43a4-a553-a7de44d4961f" (UID: "7080b36f-b381-43a4-a553-a7de44d4961f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:25.009581 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.009559 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7080b36f-b381-43a4-a553-a7de44d4961f-istio-podinfo" pod "7080b36f-b381-43a4-a553-a7de44d4961f" (UID: "7080b36f-b381-43a4-a553-a7de44d4961f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:29:25.108216 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108183 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-kube-api-access-65tlr\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-kube-api-access-65tlr\") pod \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " Oct 31 15:29:25.108402 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108305 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-credential-socket\") pod \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " Oct 31 15:29:25.108402 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108351 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-envoy\") pod \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " Oct 31 15:29:25.108545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108430 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-certs\") pod \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " Oct 31 15:29:25.108545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108461 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-token\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-token\") pod \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " Oct 31 15:29:25.108545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108490 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-podinfo\") pod \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " Oct 31 15:29:25.108723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108565 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-socket\") pod \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " Oct 31 15:29:25.108723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108598 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istiod-ca-cert\") pod \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " Oct 31 15:29:25.108723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108605 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-credential-socket" pod "5ceb65ff-52fb-40c2-ac9b-cf37657571e2" (UID: "5ceb65ff-52fb-40c2-ac9b-cf37657571e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.108723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108648 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-data\") pod \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\" (UID: \"5ceb65ff-52fb-40c2-ac9b-cf37657571e2\") " Oct 31 15:29:25.108947 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108753 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-certs" pod "5ceb65ff-52fb-40c2-ac9b-cf37657571e2" (UID: "5ceb65ff-52fb-40c2-ac9b-cf37657571e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.109020 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.108954 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-socket" pod "5ceb65ff-52fb-40c2-ac9b-cf37657571e2" (UID: "5ceb65ff-52fb-40c2-ac9b-cf37657571e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.109600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109259 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istiod-ca-cert" pod "5ceb65ff-52fb-40c2-ac9b-cf37657571e2" (UID: "5ceb65ff-52fb-40c2-ac9b-cf37657571e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:29:25.109600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109410 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7080b36f-b381-43a4-a553-a7de44d4961f-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.109600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109435 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.109600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109454 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.109600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109470 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.109600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109485 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb8q8\" (UniqueName: \"kubernetes.io/projected/7080b36f-b381-43a4-a553-a7de44d4961f-kube-api-access-vb8q8\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.109600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109502 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7080b36f-b381-43a4-a553-a7de44d4961f-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.109600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109518 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.109600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109533 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.110167 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.109685 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-data" pod "5ceb65ff-52fb-40c2-ac9b-cf37657571e2" (UID: "5ceb65ff-52fb-40c2-ac9b-cf37657571e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.111439 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.111008 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-envoy" pod "5ceb65ff-52fb-40c2-ac9b-cf37657571e2" (UID: "5ceb65ff-52fb-40c2-ac9b-cf37657571e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:29:25.111439 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.111132 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-kube-api-access-65tlr" pod "5ceb65ff-52fb-40c2-ac9b-cf37657571e2" (UID: "5ceb65ff-52fb-40c2-ac9b-cf37657571e2"). InnerVolumeSpecName "kube-api-access-65tlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:25.112222 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.112188 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-token" pod "5ceb65ff-52fb-40c2-ac9b-cf37657571e2" (UID: "5ceb65ff-52fb-40c2-ac9b-cf37657571e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:25.112545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.112520 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-podinfo" pod "5ceb65ff-52fb-40c2-ac9b-cf37657571e2" (UID: "5ceb65ff-52fb-40c2-ac9b-cf37657571e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:29:25.118409 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.118388 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:25.210630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.210599 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.210630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.210632 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.211021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.210642 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.211021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.210651 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65tlr\" (UniqueName: \"kubernetes.io/projected/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-kube-api-access-65tlr\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.211021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.210661 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ceb65ff-52fb-40c2-ac9b-cf37657571e2-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:25.499905 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.499869 4399 generic.go:356] "Generic (PLEG): container finished" podID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerID="19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5" exitCode=0 Oct 31 15:29:25.500072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.499947 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" Oct 31 15:29:25.500072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.499949 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" event={"ID":"5ceb65ff-52fb-40c2-ac9b-cf37657571e2","Type":"ContainerDied","Data":"19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5"} Oct 31 15:29:25.500072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.500002 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548" event={"ID":"5ceb65ff-52fb-40c2-ac9b-cf37657571e2","Type":"ContainerDied","Data":"56843b20a77294b6d88a905f5479b0aba44526bb2315069c8f3294305d32215c"} Oct 31 15:29:25.500072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.500025 4399 scope.go:122] "RemoveContainer" containerID="19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5" Oct 31 15:29:25.502407 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.502381 4399 generic.go:356] "Generic (PLEG): container finished" podID="7080b36f-b381-43a4-a553-a7de44d4961f" containerID="ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b" exitCode=0 Oct 31 15:29:25.502553 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.502450 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" event={"ID":"7080b36f-b381-43a4-a553-a7de44d4961f","Type":"ContainerDied","Data":"ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b"} Oct 31 15:29:25.502553 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.502492 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" event={"ID":"7080b36f-b381-43a4-a553-a7de44d4961f","Type":"ContainerDied","Data":"c3f35a7982df8bb4bd018642b0259f6c9df43c8f0b08a8b39c64d79ff597f79c"} Oct 31 15:29:25.502668 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.502563 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj" Oct 31 15:29:25.504741 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.504684 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn"] Oct 31 15:29:25.518392 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:29:25.518357 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65e4c6b_bd54_420c_b113_fb3ead82543b.slice/crio-1cdbaa24b5637a55af1d53421d658ed92b7a2a800a92b9e502e677087938900d WatchSource:0}: Error finding container 1cdbaa24b5637a55af1d53421d658ed92b7a2a800a92b9e502e677087938900d: Status 404 returned error can't find the container with id 1cdbaa24b5637a55af1d53421d658ed92b7a2a800a92b9e502e677087938900d Oct 31 15:29:25.519809 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.519787 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 15:29:25.526984 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.526966 4399 scope.go:122] "RemoveContainer" containerID="19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5" Oct 31 15:29:25.527224 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:25.527205 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5\": container with ID starting with 19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5 not found: ID does not exist" containerID="19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5" Oct 31 15:29:25.527269 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.527244 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5"} err="failed to get container status \"19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5\": rpc error: code = NotFound desc = could not find container \"19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5\": container with ID starting with 19cc6c8ced79f710192f11c42a97c5a3df4a03b90eae0124e7ffe3e7cfa5dff5 not found: ID does not exist" Oct 31 15:29:25.527313 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.527269 4399 scope.go:122] "RemoveContainer" containerID="ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b" Oct 31 15:29:25.536221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.536201 4399 scope.go:122] "RemoveContainer" containerID="ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b" Oct 31 15:29:25.536460 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:25.536442 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b\": container with ID starting with ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b not found: ID does not exist" containerID="ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b" Oct 31 15:29:25.536521 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.536468 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b"} err="failed to get container status \"ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b\": rpc error: code = NotFound desc = could not find container \"ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b\": container with ID starting with ba0025133d4d19b4b7dea78c5a8bd983c7f39a3f8d434a12fe1e77d509342d5b not found: ID does not exist" Oct 31 15:29:25.667225 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.667123 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548"] Oct 31 15:29:25.716204 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.716177 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-q8548"] Oct 31 15:29:25.778185 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.778148 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj"] Oct 31 15:29:25.781933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:25.781910 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d9cddd-openshift-default-84cf8bfb9c-hpbrj"] Oct 31 15:29:26.208682 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:26.208648 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:29:26.515994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:26.512803 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" event={"ID":"e65e4c6b-bd54-420c-b113-fb3ead82543b","Type":"ContainerStarted","Data":"fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da"} Oct 31 15:29:26.515994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:26.512891 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" event={"ID":"e65e4c6b-bd54-420c-b113-fb3ead82543b","Type":"ContainerStarted","Data":"1cdbaa24b5637a55af1d53421d658ed92b7a2a800a92b9e502e677087938900d"} Oct 31 15:29:26.515994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:26.513398 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:26.672467 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:26.672390 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" podStartSLOduration=1.839608041 podStartE2EDuration="2.672369214s" totalImagesPullingTime="832.761173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:29:26.667766526 +0000 UTC m=+2609.879007938" watchObservedRunningTime="2026-10-31 15:29:26.672369214 +0000 UTC m=+2609.883610627" Oct 31 15:29:26.897896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:26.897819 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" path="/var/lib/kubelet/pods/5ceb65ff-52fb-40c2-ac9b-cf37657571e2/volumes" Oct 31 15:29:26.898306 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:26.898292 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" path="/var/lib/kubelet/pods/7080b36f-b381-43a4-a553-a7de44d4961f/volumes" Oct 31 15:29:32.527916 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:32.527887 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 15:29:38.939532 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:38.939500 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh"] Oct 31 15:29:38.939934 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:38.939692 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" podUID="bfc30e0b-d599-4bdf-a30e-dec18a791056" containerName="maas-api" containerID="cri-o://ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5" gracePeriod=30 Oct 31 15:29:39.286852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.286823 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:29:39.399393 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.398126 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfc30e0b-d599-4bdf-a30e-dec18a791056-kube-api-access-9ck2w\" (UniqueName: \"kubernetes.io/projected/bfc30e0b-d599-4bdf-a30e-dec18a791056-kube-api-access-9ck2w\") pod \"bfc30e0b-d599-4bdf-a30e-dec18a791056\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " Oct 31 15:29:39.399393 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.398212 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfc30e0b-d599-4bdf-a30e-dec18a791056-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc30e0b-d599-4bdf-a30e-dec18a791056-openshift-service-ca\") pod \"bfc30e0b-d599-4bdf-a30e-dec18a791056\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " Oct 31 15:29:39.399393 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.398307 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-metrics-certs\") pod \"bfc30e0b-d599-4bdf-a30e-dec18a791056\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " Oct 31 15:29:39.399393 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.398459 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls\") pod \"bfc30e0b-d599-4bdf-a30e-dec18a791056\" (UID: \"bfc30e0b-d599-4bdf-a30e-dec18a791056\") " Oct 31 15:29:39.400714 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.400388 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfc30e0b-d599-4bdf-a30e-dec18a791056-openshift-service-ca" pod "bfc30e0b-d599-4bdf-a30e-dec18a791056" (UID: "bfc30e0b-d599-4bdf-a30e-dec18a791056"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:29:39.411456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.410306 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls" pod "bfc30e0b-d599-4bdf-a30e-dec18a791056" (UID: "bfc30e0b-d599-4bdf-a30e-dec18a791056"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:29:39.415685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.415635 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfc30e0b-d599-4bdf-a30e-dec18a791056-kube-api-access-9ck2w" pod "bfc30e0b-d599-4bdf-a30e-dec18a791056" (UID: "bfc30e0b-d599-4bdf-a30e-dec18a791056"). InnerVolumeSpecName "kube-api-access-9ck2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:29:39.417903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.416048 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-metrics-certs" pod "bfc30e0b-d599-4bdf-a30e-dec18a791056" (UID: "bfc30e0b-d599-4bdf-a30e-dec18a791056"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:29:39.500103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.500071 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bfc30e0b-d599-4bdf-a30e-dec18a791056-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:39.500103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.500101 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:39.500286 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.500112 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bfc30e0b-d599-4bdf-a30e-dec18a791056-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:39.500286 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.500121 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ck2w\" (UniqueName: \"kubernetes.io/projected/bfc30e0b-d599-4bdf-a30e-dec18a791056-kube-api-access-9ck2w\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:29:39.593114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.593023 4399 generic.go:356] "Generic (PLEG): container finished" podID="bfc30e0b-d599-4bdf-a30e-dec18a791056" containerID="ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5" exitCode=0 Oct 31 15:29:39.593114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.593085 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" event={"ID":"bfc30e0b-d599-4bdf-a30e-dec18a791056","Type":"ContainerDied","Data":"ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5"} Oct 31 15:29:39.593313 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.593078 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" Oct 31 15:29:39.593313 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.593129 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh" event={"ID":"bfc30e0b-d599-4bdf-a30e-dec18a791056","Type":"ContainerDied","Data":"de139770e919b2b28f6ca9c1541a08310460884f21a2213c24b69ca50c1bfb5a"} Oct 31 15:29:39.593313 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.593152 4399 scope.go:122] "RemoveContainer" containerID="ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5" Oct 31 15:29:39.603651 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.603630 4399 scope.go:122] "RemoveContainer" containerID="ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5" Oct 31 15:29:39.603923 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:39.603902 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5\": container with ID starting with ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5 not found: ID does not exist" containerID="ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5" Oct 31 15:29:39.603988 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.603934 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5"} err="failed to get container status \"ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5\": rpc error: code = NotFound desc = could not find container \"ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5\": container with ID starting with ec38bdd2e20d999ccf65fdda645916ddfcbe66c4824f476ce970b8aadd66fae5 not found: ID does not exist" Oct 31 15:29:39.755819 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.755793 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh"] Oct 31 15:29:39.758502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:39.758474 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-ebadea-765c999b7d-zlnwh"] Oct 31 15:29:40.897434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:40.897398 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfc30e0b-d599-4bdf-a30e-dec18a791056" path="/var/lib/kubelet/pods/bfc30e0b-d599-4bdf-a30e-dec18a791056/volumes" Oct 31 15:29:45.564061 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564021 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6"] Oct 31 15:29:45.564630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564612 4399 cpu_manager.go:414] "containerMap: removing container" podUID="bfc30e0b-d599-4bdf-a30e-dec18a791056" containerName="maas-api" Oct 31 15:29:45.564677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564633 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfc30e0b-d599-4bdf-a30e-dec18a791056" containerName="maas-api" Oct 31 15:29:45.564677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564659 4399 cpu_manager.go:414] "containerMap: removing container" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerName="istio-proxy" Oct 31 15:29:45.564677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564664 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerName="istio-proxy" Oct 31 15:29:45.564827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564677 4399 cpu_manager.go:414] "containerMap: removing container" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" Oct 31 15:29:45.564827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564684 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" Oct 31 15:29:45.564827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564803 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfc30e0b-d599-4bdf-a30e-dec18a791056" containerName="maas-api" Oct 31 15:29:45.564827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564814 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ceb65ff-52fb-40c2-ac9b-cf37657571e2" containerName="istio-proxy" Oct 31 15:29:45.564827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.564821 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7080b36f-b381-43a4-a553-a7de44d4961f" containerName="istio-proxy" Oct 31 15:29:45.569248 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.569229 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:45.627230 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.627196 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6"] Oct 31 15:29:45.742722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.739022 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-b25761\"" Oct 31 15:29:45.742722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.739795 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-b25761-dockercfg-6sxkh\"" Oct 31 15:29:45.742722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.741610 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf"] Oct 31 15:29:45.761742 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.761686 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:45.763253 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.763228 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59334-0191-4990-b6e2-245dda9d2f3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6\" (UID: \"a0e59334-0191-4990-b6e2-245dda9d2f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:45.763574 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.763556 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56zl\" (UniqueName: \"kubernetes.io/projected/a0e59334-0191-4990-b6e2-245dda9d2f3a-kube-api-access-n56zl\") pod \"payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6\" (UID: \"a0e59334-0191-4990-b6e2-245dda9d2f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:45.770586 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.770561 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm"] Oct 31 15:29:45.776078 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.776059 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.781545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.781492 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf"] Oct 31 15:29:45.827942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.827866 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-aa864d-serving-cert\"" Oct 31 15:29:45.844690 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.844658 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm"] Oct 31 15:29:45.864758 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.864716 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59334-0191-4990-b6e2-245dda9d2f3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6\" (UID: \"a0e59334-0191-4990-b6e2-245dda9d2f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:45.864913 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.864809 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.864967 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.864935 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jgsc\" (UniqueName: \"kubernetes.io/projected/db46a452-d867-4b8f-a021-b626034df3a3-kube-api-access-6jgsc\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.865035 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.865017 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56zl\" (UniqueName: \"kubernetes.io/projected/a0e59334-0191-4990-b6e2-245dda9d2f3a-kube-api-access-n56zl\") pod \"payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6\" (UID: \"a0e59334-0191-4990-b6e2-245dda9d2f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:45.865113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.865092 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db46a452-d867-4b8f-a021-b626034df3a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.865169 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.865118 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3999cdd-5827-4754-bc49-e7e6917f4688-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf\" (UID: \"d3999cdd-5827-4754-bc49-e7e6917f4688\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:45.865305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.865276 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvnt9\" (UniqueName: \"kubernetes.io/projected/d3999cdd-5827-4754-bc49-e7e6917f4688-kube-api-access-vvnt9\") pod \"payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf\" (UID: \"d3999cdd-5827-4754-bc49-e7e6917f4688\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:45.865427 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.865355 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.865645 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.865620 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59334-0191-4990-b6e2-245dda9d2f3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6\" (UID: \"a0e59334-0191-4990-b6e2-245dda9d2f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:45.921939 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.921906 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56zl\" (UniqueName: \"kubernetes.io/projected/a0e59334-0191-4990-b6e2-245dda9d2f3a-kube-api-access-n56zl\") pod \"payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6\" (UID: \"a0e59334-0191-4990-b6e2-245dda9d2f3a\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:45.966905 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.966870 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvnt9\" (UniqueName: \"kubernetes.io/projected/d3999cdd-5827-4754-bc49-e7e6917f4688-kube-api-access-vvnt9\") pod \"payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf\" (UID: \"d3999cdd-5827-4754-bc49-e7e6917f4688\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:45.967085 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.966933 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.967085 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.967005 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.967259 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.967081 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jgsc\" (UniqueName: \"kubernetes.io/projected/db46a452-d867-4b8f-a021-b626034df3a3-kube-api-access-6jgsc\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.967259 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.967197 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db46a452-d867-4b8f-a021-b626034df3a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.967259 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.967228 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3999cdd-5827-4754-bc49-e7e6917f4688-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf\" (UID: \"d3999cdd-5827-4754-bc49-e7e6917f4688\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:45.967553 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:45.967524 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-serving-cert: secret "maas-api-e2e-shared-b-w1-aa864d-serving-cert" not found Oct 31 15:29:45.967692 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:45.967644 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls podName:db46a452-d867-4b8f-a021-b626034df3a3 nodeName:}" failed. No retries permitted until 2026-10-31 15:29:46.467621386 +0000 UTC m=+2629.678862793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls") pod "maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" (UID: "db46a452-d867-4b8f-a021-b626034df3a3") : secret "maas-api-e2e-shared-b-w1-aa864d-serving-cert" not found Oct 31 15:29:45.968244 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.968215 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3999cdd-5827-4754-bc49-e7e6917f4688-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf\" (UID: \"d3999cdd-5827-4754-bc49-e7e6917f4688\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:45.969401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.969374 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db46a452-d867-4b8f-a021-b626034df3a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:45.971369 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:45.971342 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:46.023536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.023506 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvnt9\" (UniqueName: \"kubernetes.io/projected/d3999cdd-5827-4754-bc49-e7e6917f4688-kube-api-access-vvnt9\") pod \"payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf\" (UID: \"d3999cdd-5827-4754-bc49-e7e6917f4688\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:46.024793 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.024771 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jgsc\" (UniqueName: \"kubernetes.io/projected/db46a452-d867-4b8f-a021-b626034df3a3-kube-api-access-6jgsc\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:46.080040 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.079976 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:46.200720 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.184940 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:46.467568 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.467541 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf"] Oct 31 15:29:46.469374 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:29:46.469341 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3999cdd_5827_4754_bc49_e7e6917f4688.slice/crio-938042fea23ac60614818cdadaefe81d58db38f944f035aa5352a95e0ed25f3d WatchSource:0}: Error finding container 938042fea23ac60614818cdadaefe81d58db38f944f035aa5352a95e0ed25f3d: Status 404 returned error can't find the container with id 938042fea23ac60614818cdadaefe81d58db38f944f035aa5352a95e0ed25f3d Oct 31 15:29:46.493084 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.493059 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:46.495394 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.495368 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:46.652885 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.652856 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6"] Oct 31 15:29:46.654352 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:29:46.654325 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0e59334_0191_4990_b6e2_245dda9d2f3a.slice/crio-8aeb51d8bc6400ccb4262071300ca65746e1d96898575b2219610fd68ec92419 WatchSource:0}: Error finding container 8aeb51d8bc6400ccb4262071300ca65746e1d96898575b2219610fd68ec92419: Status 404 returned error can't find the container with id 8aeb51d8bc6400ccb4262071300ca65746e1d96898575b2219610fd68ec92419 Oct 31 15:29:46.698418 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.698386 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:46.777068 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.777021 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" event={"ID":"a0e59334-0191-4990-b6e2-245dda9d2f3a","Type":"ContainerStarted","Data":"82fc64481fe8f60857209e710766a9b823510827f3cf5e8140f15bd45300745a"} Oct 31 15:29:46.777217 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.777088 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" event={"ID":"a0e59334-0191-4990-b6e2-245dda9d2f3a","Type":"ContainerStarted","Data":"8aeb51d8bc6400ccb4262071300ca65746e1d96898575b2219610fd68ec92419"} Oct 31 15:29:46.777217 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.777117 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:46.782618 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.782569 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" event={"ID":"d3999cdd-5827-4754-bc49-e7e6917f4688","Type":"ContainerStarted","Data":"3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c"} Oct 31 15:29:46.782802 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.782632 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" event={"ID":"d3999cdd-5827-4754-bc49-e7e6917f4688","Type":"ContainerStarted","Data":"938042fea23ac60614818cdadaefe81d58db38f944f035aa5352a95e0ed25f3d"} Oct 31 15:29:46.782802 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.782687 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:46.894689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:46.894629 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" podStartSLOduration=1.894611542 podStartE2EDuration="1.894611542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:29:46.89058957 +0000 UTC m=+2630.101830984" watchObservedRunningTime="2026-10-31 15:29:46.894611542 +0000 UTC m=+2630.105852956" Oct 31 15:29:47.060391 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:47.060337 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" podStartSLOduration=2.06032218 podStartE2EDuration="2.06032218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:29:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:29:47.059363474 +0000 UTC m=+2630.270604887" watchObservedRunningTime="2026-10-31 15:29:47.06032218 +0000 UTC m=+2630.271563592" Oct 31 15:29:47.080156 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:47.080131 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm"] Oct 31 15:29:47.083201 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:29:47.083169 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb46a452_d867_4b8f_a021_b626034df3a3.slice/crio-b8656e8d69d157a285ca344b37dd0c5a7cb3ac944ff012b67d3f445aee82f47e WatchSource:0}: Error finding container b8656e8d69d157a285ca344b37dd0c5a7cb3ac944ff012b67d3f445aee82f47e: Status 404 returned error can't find the container with id b8656e8d69d157a285ca344b37dd0c5a7cb3ac944ff012b67d3f445aee82f47e Oct 31 15:29:47.790159 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:47.790110 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" event={"ID":"db46a452-d867-4b8f-a021-b626034df3a3","Type":"ContainerStarted","Data":"b8656e8d69d157a285ca344b37dd0c5a7cb3ac944ff012b67d3f445aee82f47e"} Oct 31 15:29:48.798873 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:48.798843 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" event={"ID":"db46a452-d867-4b8f-a021-b626034df3a3","Type":"ContainerStarted","Data":"90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67"} Oct 31 15:29:48.799353 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:48.798971 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:48.914885 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:48.914837 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" podStartSLOduration=3.095399604 podStartE2EDuration="3.914822554s" totalImagesPullingTime="819.42295ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:29:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:29:48.911919826 +0000 UTC m=+2632.123161240" watchObservedRunningTime="2026-10-31 15:29:48.914822554 +0000 UTC m=+2632.126063965" Oct 31 15:29:54.811202 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:54.811169 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 15:29:55.748884 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.748848 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv"] Oct 31 15:29:55.752999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.752981 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.824272 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.824231 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqvn\" (UniqueName: \"kubernetes.io/projected/75a42016-5d8e-493b-934d-fd149a21b752-kube-api-access-thqvn\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.824639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.824382 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.824639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.824510 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a42016-5d8e-493b-934d-fd149a21b752-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.824639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.824543 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.854852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.854830 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-46bf36-serving-cert\"" Oct 31 15:29:55.857969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.857946 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv"] Oct 31 15:29:55.925266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.925240 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a42016-5d8e-493b-934d-fd149a21b752-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.925417 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.925271 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.925417 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.925379 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqvn\" (UniqueName: \"kubernetes.io/projected/75a42016-5d8e-493b-934d-fd149a21b752-kube-api-access-thqvn\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.925417 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:55.925396 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-serving-cert: secret "maas-api-e2e-shared-b-w4-46bf36-serving-cert" not found Oct 31 15:29:55.925586 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.925448 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.925586 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:55.925485 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls podName:75a42016-5d8e-493b-934d-fd149a21b752 nodeName:}" failed. No retries permitted until 2026-10-31 15:29:56.425465708 +0000 UTC m=+2639.636707099 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls") pod "maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" (UID: "75a42016-5d8e-493b-934d-fd149a21b752") : secret "maas-api-e2e-shared-b-w4-46bf36-serving-cert" not found Oct 31 15:29:55.926011 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.925992 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a42016-5d8e-493b-934d-fd149a21b752-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.928163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.928138 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:55.982019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:55.981994 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqvn\" (UniqueName: \"kubernetes.io/projected/75a42016-5d8e-493b-934d-fd149a21b752-kube-api-access-thqvn\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:56.229406 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:29:56.229369 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:29:56.438525 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:56.438489 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:56.441220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:56.441188 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:56.666434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:56.666337 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:57.043565 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:57.043543 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv"] Oct 31 15:29:57.045329 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:29:57.045305 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a42016_5d8e_493b_934d_fd149a21b752.slice/crio-f6c49cdeeacd71ecc1c4c0d666a322fef47158fa773e7d68beb8a897b5dfc974 WatchSource:0}: Error finding container f6c49cdeeacd71ecc1c4c0d666a322fef47158fa773e7d68beb8a897b5dfc974: Status 404 returned error can't find the container with id f6c49cdeeacd71ecc1c4c0d666a322fef47158fa773e7d68beb8a897b5dfc974 Oct 31 15:29:57.823628 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:57.822512 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:29:57.824755 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:57.824080 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:29:57.857628 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:57.857596 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" event={"ID":"75a42016-5d8e-493b-934d-fd149a21b752","Type":"ContainerStarted","Data":"f6c49cdeeacd71ecc1c4c0d666a322fef47158fa773e7d68beb8a897b5dfc974"} Oct 31 15:29:58.865492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:58.865455 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" event={"ID":"75a42016-5d8e-493b-934d-fd149a21b752","Type":"ContainerStarted","Data":"13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98"} Oct 31 15:29:58.865983 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:58.865526 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:29:58.981767 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:29:58.981712 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" podStartSLOduration=3.161749264 podStartE2EDuration="3.981683816s" totalImagesPullingTime="819.934552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:29:58.978546016 +0000 UTC m=+2642.189787442" watchObservedRunningTime="2026-10-31 15:29:58.981683816 +0000 UTC m=+2642.192925227" Oct 31 15:30:04.876381 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:30:04.876347 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:30:26.208178 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:30:26.208143 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:30:56.207255 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:30:56.207220 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:31:08.652172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.652141 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq"] Oct 31 15:31:08.656467 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.656446 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.708379 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.708351 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-b25761-openshift-default-dockercfg-5zqmh\"" Oct 31 15:31:08.750661 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.750518 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e8733f-c3e8-4521-8a37-eae42b12481a-istiod-ca-cert\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.750661 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.750637 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-podinfo\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.750936 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.750838 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7zx5\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-kube-api-access-f7zx5\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.751152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.750951 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.751152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.750983 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-envoy\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.751152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.751013 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-credential-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.752045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.751162 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-certs\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.752045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.751203 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-data\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.752045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.751306 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-token\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.761999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.761969 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq"] Oct 31 15:31:08.852122 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852092 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-credential-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852158 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-certs\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852185 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-data\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852212 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-token\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852250 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e8733f-c3e8-4521-8a37-eae42b12481a-istiod-ca-cert\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852313 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-podinfo\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852403 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7zx5\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-kube-api-access-f7zx5\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852496 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852528 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-credential-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852520 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-envoy\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852837 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-socket\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.852982 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852961 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-certs\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.853049 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.852992 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-data\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.853116 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.853089 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e8733f-c3e8-4521-8a37-eae42b12481a-istiod-ca-cert\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.854836 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.854812 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-envoy\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.855044 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.855027 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-podinfo\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.908239 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.908181 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-token\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.908388 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.908367 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7zx5\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-kube-api-access-f7zx5\") pod \"e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:08.999675 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:08.998399 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:09.441513 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:09.441486 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq"] Oct 31 15:31:09.441806 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:31:09.441782 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e8733f_c3e8_4521_8a37_eae42b12481a.slice/crio-0aadcb404e4ae5247734b6d74a1bc2d7ea6f0593fac392233cd2ce4885ece693 WatchSource:0}: Error finding container 0aadcb404e4ae5247734b6d74a1bc2d7ea6f0593fac392233cd2ce4885ece693: Status 404 returned error can't find the container with id 0aadcb404e4ae5247734b6d74a1bc2d7ea6f0593fac392233cd2ce4885ece693 Oct 31 15:31:09.444307 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:09.444276 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:31:09.444393 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:09.444339 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:31:09.444393 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:09.444369 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:31:10.407405 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:10.407362 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" event={"ID":"04e8733f-c3e8-4521-8a37-eae42b12481a","Type":"ContainerStarted","Data":"25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03"} Oct 31 15:31:10.407835 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:10.407461 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" event={"ID":"04e8733f-c3e8-4521-8a37-eae42b12481a","Type":"ContainerStarted","Data":"0aadcb404e4ae5247734b6d74a1bc2d7ea6f0593fac392233cd2ce4885ece693"} Oct 31 15:31:10.577788 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:10.577733 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" podStartSLOduration=2.577716328 podStartE2EDuration="2.577716328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:31:10.574436783 +0000 UTC m=+2713.785678196" watchObservedRunningTime="2026-10-31 15:31:10.577716328 +0000 UTC m=+2713.788957739" Oct 31 15:31:10.998906 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:10.998869 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:11.000602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:11.000572 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:31:11.000752 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:11.000615 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 15:31:11.999822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:11.999790 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:31:12.000219 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:11.999850 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 15:31:12.999247 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:12.999210 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:31:12.999424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:12.999274 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 15:31:14.005234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:14.000102 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:31:14.005234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:14.000193 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 15:31:14.999495 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:14.999460 4399 patch_prober.go:29] interesting pod/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 31 15:31:14.999676 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:14.999514 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 31 15:31:16.004197 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:16.004164 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:16.004602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:16.004578 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:16.005405 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:16.005373 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:31:26.207600 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:31:26.207565 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:31:37.273562 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.273480 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb"] Oct 31 15:31:37.277796 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.277779 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.329641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.329620 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-2ad95a-openshift-default-dockercfg-6br7x\"" Oct 31 15:31:37.355711 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.355678 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.355829 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.355726 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.355872 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.355819 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.355919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.355874 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-974kq\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-kube-api-access-974kq\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.355990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.355971 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.356068 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.356051 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.356172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.356155 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa09f46-6095-4714-93a4-5fd3047fd141-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.356259 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.356246 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.356369 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.356352 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.383276 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.383249 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb"] Oct 31 15:31:37.457542 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.457513 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.457689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.457570 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.457689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.457587 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.457689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.457614 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.457689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.457654 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-974kq\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-kube-api-access-974kq\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.457689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.457677 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.457997 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.457756 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.457997 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.457823 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa09f46-6095-4714-93a4-5fd3047fd141-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.457997 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.457964 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.458164 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.458052 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.458164 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.458069 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.458164 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.458128 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.458330 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.458195 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.458628 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.458603 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa09f46-6095-4714-93a4-5fd3047fd141-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.460328 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.460298 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.460434 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.460350 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.514868 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.514832 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.514994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.514954 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-974kq\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-kube-api-access-974kq\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:37.592260 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:37.592191 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:38.020356 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:38.020327 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb"] Oct 31 15:31:38.021528 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:31:38.021505 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa09f46_6095_4714_93a4_5fd3047fd141.slice/crio-b9ce856fd285658e79e2b19573a1f54c1c616a22436bf15cd68c30fe964f624d WatchSource:0}: Error finding container b9ce856fd285658e79e2b19573a1f54c1c616a22436bf15cd68c30fe964f624d: Status 404 returned error can't find the container with id b9ce856fd285658e79e2b19573a1f54c1c616a22436bf15cd68c30fe964f624d Oct 31 15:31:38.024255 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:38.024222 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:31:38.024336 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:38.024284 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:31:38.024336 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:38.024313 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:31:38.578571 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:38.578532 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" event={"ID":"3fa09f46-6095-4714-93a4-5fd3047fd141","Type":"ContainerStarted","Data":"6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc"} Oct 31 15:31:38.578995 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:38.578624 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" event={"ID":"3fa09f46-6095-4714-93a4-5fd3047fd141","Type":"ContainerStarted","Data":"b9ce856fd285658e79e2b19573a1f54c1c616a22436bf15cd68c30fe964f624d"} Oct 31 15:31:38.592944 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:38.592914 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:38.699689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:38.699626 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" podStartSLOduration=1.699604752 podStartE2EDuration="1.699604752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:31:38.694824888 +0000 UTC m=+2741.906066301" watchObservedRunningTime="2026-10-31 15:31:38.699604752 +0000 UTC m=+2741.910846164" Oct 31 15:31:39.594726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:39.594680 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 15:31:39.595154 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:39.594744 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 15:31:40.592882 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:40.592849 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 15:31:40.593152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:40.592905 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 15:31:41.592621 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:41.592590 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 15:31:41.593097 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:41.592642 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 15:31:42.593470 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:42.593423 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 15:31:42.594046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:42.593477 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 15:31:43.593506 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:43.593470 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Oct 31 15:31:43.593931 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:43.593535 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Oct 31 15:31:44.597088 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:44.597060 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:44.597541 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:44.597528 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:44.598203 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:31:44.598183 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:31:56.207324 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:31:56.207287 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:32:22.620463 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.620430 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7"] Oct 31 15:32:22.625987 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.625970 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.692456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.692416 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.692613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.692466 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.692613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.692586 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.692759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.692651 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.692759 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.692751 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.692864 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.692779 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpnk8\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-kube-api-access-fpnk8\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.692864 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.692812 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.692972 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.692866 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.693044 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.693019 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.725845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.725817 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7"] Oct 31 15:32:22.794336 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794308 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.794517 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794341 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.794517 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794366 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpnk8\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-kube-api-access-fpnk8\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.794517 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794398 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.794517 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794415 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.794517 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794476 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.794855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794569 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.794855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794609 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.794855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794720 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.794855 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794827 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-credential-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.795103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794910 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-socket\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.795103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.794975 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-certs\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.795103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.795060 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-data\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.795262 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.795211 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istiod-ca-cert\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.797101 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.797078 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-envoy\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.797236 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.797205 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-podinfo\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.851502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.851477 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-token\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.851657 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.851638 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpnk8\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-kube-api-access-fpnk8\") pod \"e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:22.940300 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:22.940219 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:23.368577 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:23.368546 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7"] Oct 31 15:32:23.369202 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:32:23.369178 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fdf8f0_3dc7_4bb3_82a1_9c760f23d343.slice/crio-02bedae1c064d3564f868680b42f9ca3720b00d9a73ae9865073e9dcb120cca8 WatchSource:0}: Error finding container 02bedae1c064d3564f868680b42f9ca3720b00d9a73ae9865073e9dcb120cca8: Status 404 returned error can't find the container with id 02bedae1c064d3564f868680b42f9ca3720b00d9a73ae9865073e9dcb120cca8 Oct 31 15:32:23.371662 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:23.371628 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:32:23.371810 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:23.371724 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:32:23.371810 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:23.371774 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:32:23.882148 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:23.881017 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" event={"ID":"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343","Type":"ContainerStarted","Data":"a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2"} Oct 31 15:32:23.882148 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:23.881094 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" event={"ID":"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343","Type":"ContainerStarted","Data":"02bedae1c064d3564f868680b42f9ca3720b00d9a73ae9865073e9dcb120cca8"} Oct 31 15:32:23.942089 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:23.942050 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:24.051736 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:24.050933 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" podStartSLOduration=2.050909306 podStartE2EDuration="2.050909306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:32:23.996198621 +0000 UTC m=+2787.207440034" watchObservedRunningTime="2026-10-31 15:32:24.050909306 +0000 UTC m=+2787.262150722" Oct 31 15:32:24.945320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:24.945287 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 15:32:24.946021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:24.945976 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 15:32:25.959853 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:25.959812 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 15:32:25.960455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:25.960420 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 15:32:26.207123 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:32:26.207087 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:32:26.940588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:26.940558 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 15:32:26.940809 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:26.940604 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 15:32:27.966103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:27.966068 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 15:32:27.966596 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:27.966259 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 15:32:28.954726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:28.940736 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Oct 31 15:32:28.954989 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:28.954958 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Oct 31 15:32:29.946991 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:29.946960 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:29.947388 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:29.947359 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:29.948043 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:29.948024 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:32:42.750288 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:42.750246 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z"] Oct 31 15:32:42.750798 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:42.750511 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" containerID="cri-o://e882fbc72914c48af8f5aa32848a3c61f244d76f71006a602e8f5d7c43766ed6" gracePeriod=30 Oct 31 15:32:42.800202 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:42.800173 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2"] Oct 31 15:32:42.800416 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:42.800387 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" containerID="cri-o://13c1b5a6a66e977d3f38b288d73de52c1a7a5000f94796c89a2fff6db73fb6cf" gracePeriod=30 Oct 31 15:32:42.804207 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:42.804173 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz"] Oct 31 15:32:42.804536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:42.804496 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" containerID="cri-o://7ef9143e9369abcc72062688e24a347e44d5bc22b5c726133bfa5c4d13391012" gracePeriod=30 Oct 31 15:32:42.851333 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:42.851253 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm"] Oct 31 15:32:42.851627 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:42.851592 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" containerID="cri-o://a40cabb34c0447d7581d507ce55f166601f3a8279c45066e8b7ffcb3a56d827f" gracePeriod=30 Oct 31 15:32:47.840674 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:47.840646 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Oct 31 15:32:47.841039 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:47.840723 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Oct 31 15:32:47.927970 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:47.927938 4399 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 31 15:32:47.928089 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:47.928011 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 31 15:32:48.081595 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.080414 4399 generic.go:356] "Generic (PLEG): container finished" podID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerID="13c1b5a6a66e977d3f38b288d73de52c1a7a5000f94796c89a2fff6db73fb6cf" exitCode=0 Oct 31 15:32:48.081595 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.080518 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" event={"ID":"4683bd12-f3a9-48a0-a120-c701c479d54c","Type":"ContainerDied","Data":"13c1b5a6a66e977d3f38b288d73de52c1a7a5000f94796c89a2fff6db73fb6cf"} Oct 31 15:32:48.085042 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.085013 4399 generic.go:356] "Generic (PLEG): container finished" podID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerID="e882fbc72914c48af8f5aa32848a3c61f244d76f71006a602e8f5d7c43766ed6" exitCode=0 Oct 31 15:32:48.085320 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.085165 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" event={"ID":"cdd02a7e-dad7-4c8c-a279-93053f852bea","Type":"ContainerDied","Data":"e882fbc72914c48af8f5aa32848a3c61f244d76f71006a602e8f5d7c43766ed6"} Oct 31 15:32:48.090924 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.090453 4399 generic.go:356] "Generic (PLEG): container finished" podID="9a67bace-573f-47d7-9094-9c926e0e5300" containerID="7ef9143e9369abcc72062688e24a347e44d5bc22b5c726133bfa5c4d13391012" exitCode=0 Oct 31 15:32:48.090924 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.090556 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" event={"ID":"9a67bace-573f-47d7-9094-9c926e0e5300","Type":"ContainerDied","Data":"7ef9143e9369abcc72062688e24a347e44d5bc22b5c726133bfa5c4d13391012"} Oct 31 15:32:48.092967 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.092869 4399 generic.go:356] "Generic (PLEG): container finished" podID="070c7696-f269-4704-949a-93c33f3aac79" containerID="a40cabb34c0447d7581d507ce55f166601f3a8279c45066e8b7ffcb3a56d827f" exitCode=0 Oct 31 15:32:48.092967 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.092936 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" event={"ID":"070c7696-f269-4704-949a-93c33f3aac79","Type":"ContainerDied","Data":"a40cabb34c0447d7581d507ce55f166601f3a8279c45066e8b7ffcb3a56d827f"} Oct 31 15:32:48.094095 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.094074 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:32:48.185114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.185081 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-token\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-token\") pod \"cdd02a7e-dad7-4c8c-a279-93053f852bea\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " Oct 31 15:32:48.185274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.185126 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-credential-socket\") pod \"cdd02a7e-dad7-4c8c-a279-93053f852bea\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " Oct 31 15:32:48.185274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.185175 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-envoy\") pod \"cdd02a7e-dad7-4c8c-a279-93053f852bea\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " Oct 31 15:32:48.185274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.185233 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-kube-api-access-68dkh\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-kube-api-access-68dkh\") pod \"cdd02a7e-dad7-4c8c-a279-93053f852bea\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " Oct 31 15:32:48.185457 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.185324 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdd02a7e-dad7-4c8c-a279-93053f852bea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd02a7e-dad7-4c8c-a279-93053f852bea-istiod-ca-cert\") pod \"cdd02a7e-dad7-4c8c-a279-93053f852bea\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " Oct 31 15:32:48.185457 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.185363 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-data\") pod \"cdd02a7e-dad7-4c8c-a279-93053f852bea\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " Oct 31 15:32:48.185457 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.185426 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-podinfo\") pod \"cdd02a7e-dad7-4c8c-a279-93053f852bea\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " Oct 31 15:32:48.185753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.185608 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-socket\") pod \"cdd02a7e-dad7-4c8c-a279-93053f852bea\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " Oct 31 15:32:48.185753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.185661 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-certs\") pod \"cdd02a7e-dad7-4c8c-a279-93053f852bea\" (UID: \"cdd02a7e-dad7-4c8c-a279-93053f852bea\") " Oct 31 15:32:48.186375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.186348 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-certs" pod "cdd02a7e-dad7-4c8c-a279-93053f852bea" (UID: "cdd02a7e-dad7-4c8c-a279-93053f852bea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.187124 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.186989 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-credential-socket" pod "cdd02a7e-dad7-4c8c-a279-93053f852bea" (UID: "cdd02a7e-dad7-4c8c-a279-93053f852bea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.187909 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.187887 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdd02a7e-dad7-4c8c-a279-93053f852bea-istiod-ca-cert" pod "cdd02a7e-dad7-4c8c-a279-93053f852bea" (UID: "cdd02a7e-dad7-4c8c-a279-93053f852bea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:48.188188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.188158 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-socket" pod "cdd02a7e-dad7-4c8c-a279-93053f852bea" (UID: "cdd02a7e-dad7-4c8c-a279-93053f852bea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.188459 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.188438 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-data" pod "cdd02a7e-dad7-4c8c-a279-93053f852bea" (UID: "cdd02a7e-dad7-4c8c-a279-93053f852bea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.191375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.191351 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-envoy" pod "cdd02a7e-dad7-4c8c-a279-93053f852bea" (UID: "cdd02a7e-dad7-4c8c-a279-93053f852bea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.192524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.192297 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-kube-api-access-68dkh" pod "cdd02a7e-dad7-4c8c-a279-93053f852bea" (UID: "cdd02a7e-dad7-4c8c-a279-93053f852bea"). InnerVolumeSpecName "kube-api-access-68dkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:48.192524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.192497 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-podinfo" pod "cdd02a7e-dad7-4c8c-a279-93053f852bea" (UID: "cdd02a7e-dad7-4c8c-a279-93053f852bea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:32:48.192851 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.192829 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-token" pod "cdd02a7e-dad7-4c8c-a279-93053f852bea" (UID: "cdd02a7e-dad7-4c8c-a279-93053f852bea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:48.199519 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.199500 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:32:48.206853 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.206831 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:32:48.218182 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.218165 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:32:48.287105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287076 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-certs\") pod \"070c7696-f269-4704-949a-93c33f3aac79\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " Oct 31 15:32:48.287284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287124 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-podinfo\") pod \"4683bd12-f3a9-48a0-a120-c701c479d54c\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " Oct 31 15:32:48.287284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287151 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/070c7696-f269-4704-949a-93c33f3aac79-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070c7696-f269-4704-949a-93c33f3aac79-istio-podinfo\") pod \"070c7696-f269-4704-949a-93c33f3aac79\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " Oct 31 15:32:48.287284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287182 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-credential-socket\") pod \"070c7696-f269-4704-949a-93c33f3aac79\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " Oct 31 15:32:48.287284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287214 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-token\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-token\") pod \"4683bd12-f3a9-48a0-a120-c701c479d54c\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " Oct 31 15:32:48.287284 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287259 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-istio-token\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-istio-token\") pod \"9a67bace-573f-47d7-9094-9c926e0e5300\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " Oct 31 15:32:48.287566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287295 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-envoy\") pod \"9a67bace-573f-47d7-9094-9c926e0e5300\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " Oct 31 15:32:48.287566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287331 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-certs\") pod \"4683bd12-f3a9-48a0-a120-c701c479d54c\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " Oct 31 15:32:48.287566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287361 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-kube-api-access-kbpbr\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-kube-api-access-kbpbr\") pod \"9a67bace-573f-47d7-9094-9c926e0e5300\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " Oct 31 15:32:48.287566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287395 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9a67bace-573f-47d7-9094-9c926e0e5300-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a67bace-573f-47d7-9094-9c926e0e5300-istio-podinfo\") pod \"9a67bace-573f-47d7-9094-9c926e0e5300\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " Oct 31 15:32:48.287566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287407 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-certs" pod "070c7696-f269-4704-949a-93c33f3aac79" (UID: "070c7696-f269-4704-949a-93c33f3aac79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.287566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287426 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4683bd12-f3a9-48a0-a120-c701c479d54c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4683bd12-f3a9-48a0-a120-c701c479d54c-istiod-ca-cert\") pod \"4683bd12-f3a9-48a0-a120-c701c479d54c\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " Oct 31 15:32:48.287566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287459 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-istio-token\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-istio-token\") pod \"070c7696-f269-4704-949a-93c33f3aac79\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " Oct 31 15:32:48.287566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287497 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-data\") pod \"070c7696-f269-4704-949a-93c33f3aac79\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " Oct 31 15:32:48.287566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287543 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-socket\") pod \"070c7696-f269-4704-949a-93c33f3aac79\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287575 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/070c7696-f269-4704-949a-93c33f3aac79-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070c7696-f269-4704-949a-93c33f3aac79-istiod-ca-cert\") pod \"070c7696-f269-4704-949a-93c33f3aac79\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287618 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-envoy\") pod \"4683bd12-f3a9-48a0-a120-c701c479d54c\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287679 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-socket\") pod \"9a67bace-573f-47d7-9094-9c926e0e5300\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287743 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-kube-api-access-2rc9p\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-kube-api-access-2rc9p\") pod \"4683bd12-f3a9-48a0-a120-c701c479d54c\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287782 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-credential-socket\") pod \"4683bd12-f3a9-48a0-a120-c701c479d54c\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287813 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-socket\") pod \"4683bd12-f3a9-48a0-a120-c701c479d54c\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287847 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-certs\") pod \"9a67bace-573f-47d7-9094-9c926e0e5300\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287885 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-data\") pod \"9a67bace-573f-47d7-9094-9c926e0e5300\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.287961 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a67bace-573f-47d7-9094-9c926e0e5300-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a67bace-573f-47d7-9094-9c926e0e5300-istiod-ca-cert\") pod \"9a67bace-573f-47d7-9094-9c926e0e5300\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.288032 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-credential-socket\") pod \"9a67bace-573f-47d7-9094-9c926e0e5300\" (UID: \"9a67bace-573f-47d7-9094-9c926e0e5300\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.288133 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-kube-api-access-65jvr\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-kube-api-access-65jvr\") pod \"070c7696-f269-4704-949a-93c33f3aac79\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " Oct 31 15:32:48.288256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.288177 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-envoy\") pod \"070c7696-f269-4704-949a-93c33f3aac79\" (UID: \"070c7696-f269-4704-949a-93c33f3aac79\") " Oct 31 15:32:48.288932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.288292 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-credential-socket" pod "070c7696-f269-4704-949a-93c33f3aac79" (UID: "070c7696-f269-4704-949a-93c33f3aac79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.288932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.288725 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-data\") pod \"4683bd12-f3a9-48a0-a120-c701c479d54c\" (UID: \"4683bd12-f3a9-48a0-a120-c701c479d54c\") " Oct 31 15:32:48.289041 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.288930 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-certs" pod "9a67bace-573f-47d7-9094-9c926e0e5300" (UID: "9a67bace-573f-47d7-9094-9c926e0e5300"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.289623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289424 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-credential-socket" pod "9a67bace-573f-47d7-9094-9c926e0e5300" (UID: "9a67bace-573f-47d7-9094-9c926e0e5300"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.289748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289643 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.289748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289667 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.289748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289683 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.289748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289741 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290071 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289759 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290071 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289773 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290071 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289787 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290071 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289800 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68dkh\" (UniqueName: \"kubernetes.io/projected/cdd02a7e-dad7-4c8c-a279-93053f852bea-kube-api-access-68dkh\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290071 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289815 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290071 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289829 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cdd02a7e-dad7-4c8c-a279-93053f852bea-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290071 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289841 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290071 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289854 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cdd02a7e-dad7-4c8c-a279-93053f852bea-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290071 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.289867 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.290550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.290086 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-data" pod "4683bd12-f3a9-48a0-a120-c701c479d54c" (UID: "4683bd12-f3a9-48a0-a120-c701c479d54c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.290550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.290135 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-certs" pod "4683bd12-f3a9-48a0-a120-c701c479d54c" (UID: "4683bd12-f3a9-48a0-a120-c701c479d54c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.290550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.290172 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-socket" pod "9a67bace-573f-47d7-9094-9c926e0e5300" (UID: "9a67bace-573f-47d7-9094-9c926e0e5300"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.290550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.290381 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-credential-socket" pod "4683bd12-f3a9-48a0-a120-c701c479d54c" (UID: "4683bd12-f3a9-48a0-a120-c701c479d54c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.291456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.290771 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-socket" pod "4683bd12-f3a9-48a0-a120-c701c479d54c" (UID: "4683bd12-f3a9-48a0-a120-c701c479d54c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.291456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.290785 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-data" pod "9a67bace-573f-47d7-9094-9c926e0e5300" (UID: "9a67bace-573f-47d7-9094-9c926e0e5300"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.291456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.290890 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a67bace-573f-47d7-9094-9c926e0e5300-istiod-ca-cert" pod "9a67bace-573f-47d7-9094-9c926e0e5300" (UID: "9a67bace-573f-47d7-9094-9c926e0e5300"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:48.291456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.291027 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-data" pod "070c7696-f269-4704-949a-93c33f3aac79" (UID: "070c7696-f269-4704-949a-93c33f3aac79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.291456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.291184 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4683bd12-f3a9-48a0-a120-c701c479d54c-istiod-ca-cert" pod "4683bd12-f3a9-48a0-a120-c701c479d54c" (UID: "4683bd12-f3a9-48a0-a120-c701c479d54c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:48.291456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.291423 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-socket" pod "070c7696-f269-4704-949a-93c33f3aac79" (UID: "070c7696-f269-4704-949a-93c33f3aac79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.291940 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.291817 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/070c7696-f269-4704-949a-93c33f3aac79-istiod-ca-cert" pod "070c7696-f269-4704-949a-93c33f3aac79" (UID: "070c7696-f269-4704-949a-93c33f3aac79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:32:48.292647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.292612 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-podinfo" pod "4683bd12-f3a9-48a0-a120-c701c479d54c" (UID: "4683bd12-f3a9-48a0-a120-c701c479d54c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:32:48.293065 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.293033 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-kube-api-access-kbpbr" pod "9a67bace-573f-47d7-9094-9c926e0e5300" (UID: "9a67bace-573f-47d7-9094-9c926e0e5300"). InnerVolumeSpecName "kube-api-access-kbpbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:48.293285 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.293252 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-token" pod "4683bd12-f3a9-48a0-a120-c701c479d54c" (UID: "4683bd12-f3a9-48a0-a120-c701c479d54c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:48.293502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.293476 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/070c7696-f269-4704-949a-93c33f3aac79-istio-podinfo" pod "070c7696-f269-4704-949a-93c33f3aac79" (UID: "070c7696-f269-4704-949a-93c33f3aac79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:32:48.293663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.293593 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-envoy" pod "4683bd12-f3a9-48a0-a120-c701c479d54c" (UID: "4683bd12-f3a9-48a0-a120-c701c479d54c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.293918 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.293690 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-envoy" pod "070c7696-f269-4704-949a-93c33f3aac79" (UID: "070c7696-f269-4704-949a-93c33f3aac79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.294013 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.293987 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-kube-api-access-2rc9p" pod "4683bd12-f3a9-48a0-a120-c701c479d54c" (UID: "4683bd12-f3a9-48a0-a120-c701c479d54c"). InnerVolumeSpecName "kube-api-access-2rc9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:48.294128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.294014 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-istio-token" pod "9a67bace-573f-47d7-9094-9c926e0e5300" (UID: "9a67bace-573f-47d7-9094-9c926e0e5300"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:48.294617 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.294597 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-envoy" pod "9a67bace-573f-47d7-9094-9c926e0e5300" (UID: "9a67bace-573f-47d7-9094-9c926e0e5300"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:32:48.295055 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.295034 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-kube-api-access-65jvr" pod "070c7696-f269-4704-949a-93c33f3aac79" (UID: "070c7696-f269-4704-949a-93c33f3aac79"). InnerVolumeSpecName "kube-api-access-65jvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:48.295147 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.295072 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-istio-token" pod "070c7696-f269-4704-949a-93c33f3aac79" (UID: "070c7696-f269-4704-949a-93c33f3aac79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:32:48.295216 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.295202 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a67bace-573f-47d7-9094-9c926e0e5300-istio-podinfo" pod "9a67bace-573f-47d7-9094-9c926e0e5300" (UID: "9a67bace-573f-47d7-9094-9c926e0e5300"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:32:48.391293 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391210 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a67bace-573f-47d7-9094-9c926e0e5300-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391293 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391246 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65jvr\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-kube-api-access-65jvr\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391293 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391258 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391293 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391269 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391293 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391283 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391303 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/070c7696-f269-4704-949a-93c33f3aac79-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391311 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391320 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391329 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391337 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391350 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpbr\" (UniqueName: \"kubernetes.io/projected/9a67bace-573f-47d7-9094-9c926e0e5300-kube-api-access-kbpbr\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391364 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a67bace-573f-47d7-9094-9c926e0e5300-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391378 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4683bd12-f3a9-48a0-a120-c701c479d54c-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391393 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/070c7696-f269-4704-949a-93c33f3aac79-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391406 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391416 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/070c7696-f269-4704-949a-93c33f3aac79-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391424 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/070c7696-f269-4704-949a-93c33f3aac79-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391432 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391443 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391457 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rc9p\" (UniqueName: \"kubernetes.io/projected/4683bd12-f3a9-48a0-a120-c701c479d54c-kube-api-access-2rc9p\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391471 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391483 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4683bd12-f3a9-48a0-a120-c701c479d54c-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:48.391649 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:48.391497 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a67bace-573f-47d7-9094-9c926e0e5300-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:32:49.101031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.100956 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" Oct 31 15:32:49.101031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.100963 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2" event={"ID":"4683bd12-f3a9-48a0-a120-c701c479d54c","Type":"ContainerDied","Data":"e22db0503aca7f21a2cd6a7172b2b4754601f3a34f348448a1139b8950313002"} Oct 31 15:32:49.101031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.101027 4399 scope.go:122] "RemoveContainer" containerID="13c1b5a6a66e977d3f38b288d73de52c1a7a5000f94796c89a2fff6db73fb6cf" Oct 31 15:32:49.102496 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.102474 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" Oct 31 15:32:49.102623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.102498 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z" event={"ID":"cdd02a7e-dad7-4c8c-a279-93053f852bea","Type":"ContainerDied","Data":"7f25ebf3433e4249a3440252855600b3e4a1b35051e81993029fe11121866992"} Oct 31 15:32:49.104494 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.104472 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" Oct 31 15:32:49.104494 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.104479 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz" event={"ID":"9a67bace-573f-47d7-9094-9c926e0e5300","Type":"ContainerDied","Data":"75ffbbb495bf713ce416dc2c03cdb4b55575f5e66a6a50999a0f83fded354494"} Oct 31 15:32:49.106079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.106059 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" event={"ID":"070c7696-f269-4704-949a-93c33f3aac79","Type":"ContainerDied","Data":"a53e133b046a92fb744047738ba346e945d52135efcdbb57a327055cc0125061"} Oct 31 15:32:49.106203 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.106067 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm" Oct 31 15:32:49.113453 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.113432 4399 scope.go:122] "RemoveContainer" containerID="e882fbc72914c48af8f5aa32848a3c61f244d76f71006a602e8f5d7c43766ed6" Oct 31 15:32:49.124642 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.124621 4399 scope.go:122] "RemoveContainer" containerID="7ef9143e9369abcc72062688e24a347e44d5bc22b5c726133bfa5c4d13391012" Oct 31 15:32:49.133753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.133681 4399 scope.go:122] "RemoveContainer" containerID="a40cabb34c0447d7581d507ce55f166601f3a8279c45066e8b7ffcb3a56d827f" Oct 31 15:32:49.264492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.264468 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz"] Oct 31 15:32:49.313937 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.313910 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-2fkxz"] Oct 31 15:32:49.379061 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.378988 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm"] Oct 31 15:32:49.384613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.384591 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-gs4mm"] Oct 31 15:32:49.492234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.492192 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2"] Oct 31 15:32:49.495890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.495867 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-28gs2"] Oct 31 15:32:49.603411 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.603382 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z"] Oct 31 15:32:49.610274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:49.610249 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2348-openshift-default-74559cb757-q225z"] Oct 31 15:32:50.903660 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:50.903617 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="070c7696-f269-4704-949a-93c33f3aac79" path="/var/lib/kubelet/pods/070c7696-f269-4704-949a-93c33f3aac79/volumes" Oct 31 15:32:50.904458 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:50.904432 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" path="/var/lib/kubelet/pods/4683bd12-f3a9-48a0-a120-c701c479d54c/volumes" Oct 31 15:32:50.905501 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:50.905481 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" path="/var/lib/kubelet/pods/9a67bace-573f-47d7-9094-9c926e0e5300/volumes" Oct 31 15:32:50.906248 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:50.906232 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" path="/var/lib/kubelet/pods/cdd02a7e-dad7-4c8c-a279-93053f852bea/volumes" Oct 31 15:32:54.736634 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:54.736597 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz"] Oct 31 15:32:54.737271 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:32:54.736826 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" containerID="cri-o://6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1" gracePeriod=30 Oct 31 15:32:56.207846 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:32:56.207808 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:33:00.001310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.001286 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:33:00.156937 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.156858 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/793b21ab-9740-45db-8700-b9d5d9809d44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b21ab-9740-45db-8700-b9d5d9809d44-istio-podinfo\") pod \"793b21ab-9740-45db-8700-b9d5d9809d44\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " Oct 31 15:33:00.156937 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.156892 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-certs\") pod \"793b21ab-9740-45db-8700-b9d5d9809d44\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " Oct 31 15:33:00.157178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.156991 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-envoy\") pod \"793b21ab-9740-45db-8700-b9d5d9809d44\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " Oct 31 15:33:00.157178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157047 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-kube-api-access-9pbqj\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-kube-api-access-9pbqj\") pod \"793b21ab-9740-45db-8700-b9d5d9809d44\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " Oct 31 15:33:00.157178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157134 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/793b21ab-9740-45db-8700-b9d5d9809d44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b21ab-9740-45db-8700-b9d5d9809d44-istiod-ca-cert\") pod \"793b21ab-9740-45db-8700-b9d5d9809d44\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " Oct 31 15:33:00.157178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157159 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-istio-token\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-istio-token\") pod \"793b21ab-9740-45db-8700-b9d5d9809d44\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " Oct 31 15:33:00.157397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157191 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-data\") pod \"793b21ab-9740-45db-8700-b9d5d9809d44\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " Oct 31 15:33:00.157397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157223 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-certs" pod "793b21ab-9740-45db-8700-b9d5d9809d44" (UID: "793b21ab-9740-45db-8700-b9d5d9809d44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:00.157397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157249 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-credential-socket\") pod \"793b21ab-9740-45db-8700-b9d5d9809d44\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " Oct 31 15:33:00.157397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157325 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-socket\") pod \"793b21ab-9740-45db-8700-b9d5d9809d44\" (UID: \"793b21ab-9740-45db-8700-b9d5d9809d44\") " Oct 31 15:33:00.157621 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157491 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-data" pod "793b21ab-9740-45db-8700-b9d5d9809d44" (UID: "793b21ab-9740-45db-8700-b9d5d9809d44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:00.157621 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157557 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-credential-socket" pod "793b21ab-9740-45db-8700-b9d5d9809d44" (UID: "793b21ab-9740-45db-8700-b9d5d9809d44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:00.157794 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157689 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-socket" pod "793b21ab-9740-45db-8700-b9d5d9809d44" (UID: "793b21ab-9740-45db-8700-b9d5d9809d44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:00.157858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.157806 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/793b21ab-9740-45db-8700-b9d5d9809d44-istiod-ca-cert" pod "793b21ab-9740-45db-8700-b9d5d9809d44" (UID: "793b21ab-9740-45db-8700-b9d5d9809d44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:33:00.158315 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.158288 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/793b21ab-9740-45db-8700-b9d5d9809d44-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:00.158425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.158317 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:00.158425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.158336 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:00.158425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.158352 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:00.158425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.158367 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:00.159583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.159561 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-istio-token" pod "793b21ab-9740-45db-8700-b9d5d9809d44" (UID: "793b21ab-9740-45db-8700-b9d5d9809d44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:00.159713 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.159612 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-envoy" pod "793b21ab-9740-45db-8700-b9d5d9809d44" (UID: "793b21ab-9740-45db-8700-b9d5d9809d44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:33:00.159764 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.159692 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/793b21ab-9740-45db-8700-b9d5d9809d44-istio-podinfo" pod "793b21ab-9740-45db-8700-b9d5d9809d44" (UID: "793b21ab-9740-45db-8700-b9d5d9809d44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:33:00.159841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.159816 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-kube-api-access-9pbqj" pod "793b21ab-9740-45db-8700-b9d5d9809d44" (UID: "793b21ab-9740-45db-8700-b9d5d9809d44"). InnerVolumeSpecName "kube-api-access-9pbqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:00.188560 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.188528 4399 generic.go:356] "Generic (PLEG): container finished" podID="793b21ab-9740-45db-8700-b9d5d9809d44" containerID="6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1" exitCode=0 Oct 31 15:33:00.188929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.188654 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" Oct 31 15:33:00.189057 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.188689 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" event={"ID":"793b21ab-9740-45db-8700-b9d5d9809d44","Type":"ContainerDied","Data":"6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1"} Oct 31 15:33:00.189138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.189107 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz" event={"ID":"793b21ab-9740-45db-8700-b9d5d9809d44","Type":"ContainerDied","Data":"50444354509a021f8e434ac5d8b4f9d3770d3d79a1553f9110d2c072a868290d"} Oct 31 15:33:00.189205 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.189162 4399 scope.go:122] "RemoveContainer" containerID="6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1" Oct 31 15:33:00.202099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.202075 4399 scope.go:122] "RemoveContainer" containerID="6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1" Oct 31 15:33:00.202480 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:33:00.202452 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1\": container with ID starting with 6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1 not found: ID does not exist" containerID="6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1" Oct 31 15:33:00.202571 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.202490 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1"} err="failed to get container status \"6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1\": rpc error: code = NotFound desc = could not find container \"6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1\": container with ID starting with 6cb85658d2a2d657697d55c6aa45a6499afbbc653d46082f8900d0999c4348c1 not found: ID does not exist" Oct 31 15:33:00.259826 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.259800 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/793b21ab-9740-45db-8700-b9d5d9809d44-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:00.259826 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.259825 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbqj\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-kube-api-access-9pbqj\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:00.259999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.259837 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/793b21ab-9740-45db-8700-b9d5d9809d44-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:00.259999 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.259847 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/793b21ab-9740-45db-8700-b9d5d9809d44-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:00.354968 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.354933 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz"] Oct 31 15:33:00.358077 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.358050 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-lblwz"] Oct 31 15:33:00.897010 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:00.896974 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" path="/var/lib/kubelet/pods/793b21ab-9740-45db-8700-b9d5d9809d44/volumes" Oct 31 15:33:20.026838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:20.026800 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf"] Oct 31 15:33:20.136187 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:20.136147 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6"] Oct 31 15:33:25.027485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.027423 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" podUID="d3999cdd-5827-4754-bc49-e7e6917f4688" containerName="payload-processing" containerID="cri-o://3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c" gracePeriod=25 Oct 31 15:33:25.136688 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.136638 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" podUID="a0e59334-0191-4990-b6e2-245dda9d2f3a" containerName="payload-pre-processing" containerID="cri-o://82fc64481fe8f60857209e710766a9b823510827f3cf5e8140f15bd45300745a" gracePeriod=25 Oct 31 15:33:25.310526 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.310494 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:33:25.341593 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.341565 4399 generic.go:356] "Generic (PLEG): container finished" podID="a0e59334-0191-4990-b6e2-245dda9d2f3a" containerID="82fc64481fe8f60857209e710766a9b823510827f3cf5e8140f15bd45300745a" exitCode=0 Oct 31 15:33:25.341752 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.341638 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" event={"ID":"a0e59334-0191-4990-b6e2-245dda9d2f3a","Type":"ContainerDied","Data":"82fc64481fe8f60857209e710766a9b823510827f3cf5e8140f15bd45300745a"} Oct 31 15:33:25.343302 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.343279 4399 generic.go:356] "Generic (PLEG): container finished" podID="d3999cdd-5827-4754-bc49-e7e6917f4688" containerID="3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c" exitCode=0 Oct 31 15:33:25.343414 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.343326 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" Oct 31 15:33:25.343414 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.343359 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" event={"ID":"d3999cdd-5827-4754-bc49-e7e6917f4688","Type":"ContainerDied","Data":"3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c"} Oct 31 15:33:25.343528 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.343410 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf" event={"ID":"d3999cdd-5827-4754-bc49-e7e6917f4688","Type":"ContainerDied","Data":"938042fea23ac60614818cdadaefe81d58db38f944f035aa5352a95e0ed25f3d"} Oct 31 15:33:25.343528 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.343440 4399 scope.go:122] "RemoveContainer" containerID="3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c" Oct 31 15:33:25.354980 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.354959 4399 scope.go:122] "RemoveContainer" containerID="3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c" Oct 31 15:33:25.355271 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:33:25.355255 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c\": container with ID starting with 3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c not found: ID does not exist" containerID="3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c" Oct 31 15:33:25.355347 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.355283 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c"} err="failed to get container status \"3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c\": rpc error: code = NotFound desc = could not find container \"3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c\": container with ID starting with 3c347725581f7815c6adba2dca273973848de340634f8315d9a62040ad4d8e9c not found: ID does not exist" Oct 31 15:33:25.379509 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.379473 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3999cdd-5827-4754-bc49-e7e6917f4688-kube-api-access-vvnt9\" (UniqueName: \"kubernetes.io/projected/d3999cdd-5827-4754-bc49-e7e6917f4688-kube-api-access-vvnt9\") pod \"d3999cdd-5827-4754-bc49-e7e6917f4688\" (UID: \"d3999cdd-5827-4754-bc49-e7e6917f4688\") " Oct 31 15:33:25.379638 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.379587 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3999cdd-5827-4754-bc49-e7e6917f4688-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3999cdd-5827-4754-bc49-e7e6917f4688-plugins-config-volume\") pod \"d3999cdd-5827-4754-bc49-e7e6917f4688\" (UID: \"d3999cdd-5827-4754-bc49-e7e6917f4688\") " Oct 31 15:33:25.379970 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.379943 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3999cdd-5827-4754-bc49-e7e6917f4688-plugins-config-volume" pod "d3999cdd-5827-4754-bc49-e7e6917f4688" (UID: "d3999cdd-5827-4754-bc49-e7e6917f4688"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:33:25.380209 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.380194 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3999cdd-5827-4754-bc49-e7e6917f4688-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.382350 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.382322 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3999cdd-5827-4754-bc49-e7e6917f4688-kube-api-access-vvnt9" pod "d3999cdd-5827-4754-bc49-e7e6917f4688" (UID: "d3999cdd-5827-4754-bc49-e7e6917f4688"). InnerVolumeSpecName "kube-api-access-vvnt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:25.409330 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.409310 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:33:25.481141 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.481116 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0e59334-0191-4990-b6e2-245dda9d2f3a-kube-api-access-n56zl\" (UniqueName: \"kubernetes.io/projected/a0e59334-0191-4990-b6e2-245dda9d2f3a-kube-api-access-n56zl\") pod \"a0e59334-0191-4990-b6e2-245dda9d2f3a\" (UID: \"a0e59334-0191-4990-b6e2-245dda9d2f3a\") " Oct 31 15:33:25.481323 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.481161 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0e59334-0191-4990-b6e2-245dda9d2f3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59334-0191-4990-b6e2-245dda9d2f3a-plugins-config-volume\") pod \"a0e59334-0191-4990-b6e2-245dda9d2f3a\" (UID: \"a0e59334-0191-4990-b6e2-245dda9d2f3a\") " Oct 31 15:33:25.481485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.481470 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvnt9\" (UniqueName: \"kubernetes.io/projected/d3999cdd-5827-4754-bc49-e7e6917f4688-kube-api-access-vvnt9\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.481556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.481523 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0e59334-0191-4990-b6e2-245dda9d2f3a-plugins-config-volume" pod "a0e59334-0191-4990-b6e2-245dda9d2f3a" (UID: "a0e59334-0191-4990-b6e2-245dda9d2f3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:33:25.483520 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.483498 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0e59334-0191-4990-b6e2-245dda9d2f3a-kube-api-access-n56zl" pod "a0e59334-0191-4990-b6e2-245dda9d2f3a" (UID: "a0e59334-0191-4990-b6e2-245dda9d2f3a"). InnerVolumeSpecName "kube-api-access-n56zl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:33:25.583182 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.583104 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56zl\" (UniqueName: \"kubernetes.io/projected/a0e59334-0191-4990-b6e2-245dda9d2f3a-kube-api-access-n56zl\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.583182 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.583135 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0e59334-0191-4990-b6e2-245dda9d2f3a-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:33:25.763230 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.763192 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf"] Oct 31 15:33:25.768455 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:25.768422 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-b25761-74bcf8d4d7-xtkxf"] Oct 31 15:33:26.207756 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:33:26.207716 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:33:26.351381 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:26.351305 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" Oct 31 15:33:26.351550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:26.351306 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6" event={"ID":"a0e59334-0191-4990-b6e2-245dda9d2f3a","Type":"ContainerDied","Data":"8aeb51d8bc6400ccb4262071300ca65746e1d96898575b2219610fd68ec92419"} Oct 31 15:33:26.351550 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:26.351439 4399 scope.go:122] "RemoveContainer" containerID="82fc64481fe8f60857209e710766a9b823510827f3cf5e8140f15bd45300745a" Oct 31 15:33:26.580739 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:26.580682 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6"] Oct 31 15:33:26.583211 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:26.583192 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-b25761-b6c86c567-gdnx6"] Oct 31 15:33:26.897487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:26.897453 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0e59334-0191-4990-b6e2-245dda9d2f3a" path="/var/lib/kubelet/pods/a0e59334-0191-4990-b6e2-245dda9d2f3a/volumes" Oct 31 15:33:26.897984 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:33:26.897961 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3999cdd-5827-4754-bc49-e7e6917f4688" path="/var/lib/kubelet/pods/d3999cdd-5827-4754-bc49-e7e6917f4688/volumes" Oct 31 15:33:56.205986 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:33:56.205945 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:34:17.665194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:17.665158 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq"] Oct 31 15:34:17.665720 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:17.665395 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerName="istio-proxy" containerID="cri-o://25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03" gracePeriod=30 Oct 31 15:34:22.939027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:22.938972 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:34:23.023143 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023114 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-podinfo\") pod \"04e8733f-c3e8-4521-8a37-eae42b12481a\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " Oct 31 15:34:23.023304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023227 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e8733f-c3e8-4521-8a37-eae42b12481a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e8733f-c3e8-4521-8a37-eae42b12481a-istiod-ca-cert\") pod \"04e8733f-c3e8-4521-8a37-eae42b12481a\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " Oct 31 15:34:23.023304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023261 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-envoy\") pod \"04e8733f-c3e8-4521-8a37-eae42b12481a\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " Oct 31 15:34:23.023429 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023318 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-kube-api-access-f7zx5\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-kube-api-access-f7zx5\") pod \"04e8733f-c3e8-4521-8a37-eae42b12481a\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " Oct 31 15:34:23.023429 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023359 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-socket\") pod \"04e8733f-c3e8-4521-8a37-eae42b12481a\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " Oct 31 15:34:23.023429 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023387 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-credential-socket\") pod \"04e8733f-c3e8-4521-8a37-eae42b12481a\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " Oct 31 15:34:23.023647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023481 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-data\") pod \"04e8733f-c3e8-4521-8a37-eae42b12481a\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " Oct 31 15:34:23.023647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023537 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-token\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-token\") pod \"04e8733f-c3e8-4521-8a37-eae42b12481a\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " Oct 31 15:34:23.023803 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023745 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-socket" pod "04e8733f-c3e8-4521-8a37-eae42b12481a" (UID: "04e8733f-c3e8-4521-8a37-eae42b12481a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.023803 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023784 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-credential-socket" pod "04e8733f-c3e8-4521-8a37-eae42b12481a" (UID: "04e8733f-c3e8-4521-8a37-eae42b12481a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.023918 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023768 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e8733f-c3e8-4521-8a37-eae42b12481a-istiod-ca-cert" pod "04e8733f-c3e8-4521-8a37-eae42b12481a" (UID: "04e8733f-c3e8-4521-8a37-eae42b12481a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:34:23.023918 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023812 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-data" pod "04e8733f-c3e8-4521-8a37-eae42b12481a" (UID: "04e8733f-c3e8-4521-8a37-eae42b12481a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.024021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.023947 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-certs\") pod \"04e8733f-c3e8-4521-8a37-eae42b12481a\" (UID: \"04e8733f-c3e8-4521-8a37-eae42b12481a\") " Oct 31 15:34:23.024261 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.024233 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-certs" pod "04e8733f-c3e8-4521-8a37-eae42b12481a" (UID: "04e8733f-c3e8-4521-8a37-eae42b12481a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.024487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.024471 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e8733f-c3e8-4521-8a37-eae42b12481a-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.024487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.024489 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.024487 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.024498 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.024735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.024512 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.024735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.024521 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.025727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.025688 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-envoy" pod "04e8733f-c3e8-4521-8a37-eae42b12481a" (UID: "04e8733f-c3e8-4521-8a37-eae42b12481a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:34:23.026019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.026000 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-token" pod "04e8733f-c3e8-4521-8a37-eae42b12481a" (UID: "04e8733f-c3e8-4521-8a37-eae42b12481a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:34:23.026093 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.026071 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-podinfo" pod "04e8733f-c3e8-4521-8a37-eae42b12481a" (UID: "04e8733f-c3e8-4521-8a37-eae42b12481a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:34:23.026172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.026154 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-kube-api-access-f7zx5" pod "04e8733f-c3e8-4521-8a37-eae42b12481a" (UID: "04e8733f-c3e8-4521-8a37-eae42b12481a"). InnerVolumeSpecName "kube-api-access-f7zx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:34:23.125880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.125845 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7zx5\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-kube-api-access-f7zx5\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.125880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.125871 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.125880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.125881 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.125880 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.125889 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e8733f-c3e8-4521-8a37-eae42b12481a-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:34:23.712959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.712927 4399 generic.go:356] "Generic (PLEG): container finished" podID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerID="25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03" exitCode=0 Oct 31 15:34:23.713157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.712985 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" Oct 31 15:34:23.713157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.713012 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" event={"ID":"04e8733f-c3e8-4521-8a37-eae42b12481a","Type":"ContainerDied","Data":"25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03"} Oct 31 15:34:23.713157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.713070 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq" event={"ID":"04e8733f-c3e8-4521-8a37-eae42b12481a","Type":"ContainerDied","Data":"0aadcb404e4ae5247734b6d74a1bc2d7ea6f0593fac392233cd2ce4885ece693"} Oct 31 15:34:23.713157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.713101 4399 scope.go:122] "RemoveContainer" containerID="25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03" Oct 31 15:34:23.723947 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.723929 4399 scope.go:122] "RemoveContainer" containerID="25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03" Oct 31 15:34:23.724221 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:34:23.724203 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03\": container with ID starting with 25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03 not found: ID does not exist" containerID="25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03" Oct 31 15:34:23.724272 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.724229 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03"} err="failed to get container status \"25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03\": rpc error: code = NotFound desc = could not find container \"25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03\": container with ID starting with 25233e570e5ea7a1feaf8ad66dc7dfa6840c036dda26a511576596ac15995d03 not found: ID does not exist" Oct 31 15:34:23.926624 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.926593 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq"] Oct 31 15:34:23.976095 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:23.976030 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-b25761-openshift-default-f459fdd45-d24tq"] Oct 31 15:34:24.897343 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:34:24.897310 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" path="/var/lib/kubelet/pods/04e8733f-c3e8-4521-8a37-eae42b12481a/volumes" Oct 31 15:34:26.207427 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:34:26.207384 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:34:56.206912 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:34:56.206875 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:35:10.802179 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:10.802145 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n"] Oct 31 15:35:10.802718 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:10.802343 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" containerID="cri-o://85717764815c836dd83d89c6c58adce858f3929e9d32f2daa78b3c11edd6a0fb" gracePeriod=30 Oct 31 15:35:14.653588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:14.653552 4399 patch_prober.go:29] interesting pod/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:35:14.654008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:14.653619 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:35:16.069595 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.069518 4399 generic.go:356] "Generic (PLEG): container finished" podID="e074092c-d612-4e28-a501-ca34eab4a14f" containerID="85717764815c836dd83d89c6c58adce858f3929e9d32f2daa78b3c11edd6a0fb" exitCode=0 Oct 31 15:35:16.069595 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.069581 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" event={"ID":"e074092c-d612-4e28-a501-ca34eab4a14f","Type":"ContainerDied","Data":"85717764815c836dd83d89c6c58adce858f3929e9d32f2daa78b3c11edd6a0fb"} Oct 31 15:35:16.095371 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.095348 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:35:16.238395 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238361 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-data\") pod \"e074092c-d612-4e28-a501-ca34eab4a14f\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " Oct 31 15:35:16.238588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238409 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-socket\") pod \"e074092c-d612-4e28-a501-ca34eab4a14f\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " Oct 31 15:35:16.238588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238462 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-istio-token\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-istio-token\") pod \"e074092c-d612-4e28-a501-ca34eab4a14f\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " Oct 31 15:35:16.238588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238503 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e074092c-d612-4e28-a501-ca34eab4a14f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e074092c-d612-4e28-a501-ca34eab4a14f-istio-podinfo\") pod \"e074092c-d612-4e28-a501-ca34eab4a14f\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " Oct 31 15:35:16.238804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238619 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-envoy\") pod \"e074092c-d612-4e28-a501-ca34eab4a14f\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " Oct 31 15:35:16.238804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238649 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-data" pod "e074092c-d612-4e28-a501-ca34eab4a14f" (UID: "e074092c-d612-4e28-a501-ca34eab4a14f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:16.238804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238660 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-kube-api-access-kpbmz\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-kube-api-access-kpbmz\") pod \"e074092c-d612-4e28-a501-ca34eab4a14f\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " Oct 31 15:35:16.238804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238741 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-certs\") pod \"e074092c-d612-4e28-a501-ca34eab4a14f\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " Oct 31 15:35:16.238804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238766 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e074092c-d612-4e28-a501-ca34eab4a14f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e074092c-d612-4e28-a501-ca34eab4a14f-istiod-ca-cert\") pod \"e074092c-d612-4e28-a501-ca34eab4a14f\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " Oct 31 15:35:16.238804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238773 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-socket" pod "e074092c-d612-4e28-a501-ca34eab4a14f" (UID: "e074092c-d612-4e28-a501-ca34eab4a14f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:16.239113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.238940 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-credential-socket\") pod \"e074092c-d612-4e28-a501-ca34eab4a14f\" (UID: \"e074092c-d612-4e28-a501-ca34eab4a14f\") " Oct 31 15:35:16.239113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.239056 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-certs" pod "e074092c-d612-4e28-a501-ca34eab4a14f" (UID: "e074092c-d612-4e28-a501-ca34eab4a14f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:16.239229 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.239160 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e074092c-d612-4e28-a501-ca34eab4a14f-istiod-ca-cert" pod "e074092c-d612-4e28-a501-ca34eab4a14f" (UID: "e074092c-d612-4e28-a501-ca34eab4a14f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:35:16.239622 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.239596 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-credential-socket" pod "e074092c-d612-4e28-a501-ca34eab4a14f" (UID: "e074092c-d612-4e28-a501-ca34eab4a14f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:16.239862 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.239840 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:16.239943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.239868 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:16.239943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.239884 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:16.239943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.239901 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:16.239943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.239917 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e074092c-d612-4e28-a501-ca34eab4a14f-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:16.240990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.240964 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-envoy" pod "e074092c-d612-4e28-a501-ca34eab4a14f" (UID: "e074092c-d612-4e28-a501-ca34eab4a14f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:35:16.241091 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.240988 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e074092c-d612-4e28-a501-ca34eab4a14f-istio-podinfo" pod "e074092c-d612-4e28-a501-ca34eab4a14f" (UID: "e074092c-d612-4e28-a501-ca34eab4a14f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:35:16.241144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.241130 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-kube-api-access-kpbmz" pod "e074092c-d612-4e28-a501-ca34eab4a14f" (UID: "e074092c-d612-4e28-a501-ca34eab4a14f"). InnerVolumeSpecName "kube-api-access-kpbmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:35:16.241205 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.241184 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-istio-token" pod "e074092c-d612-4e28-a501-ca34eab4a14f" (UID: "e074092c-d612-4e28-a501-ca34eab4a14f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:35:16.341490 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.341414 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:16.341490 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.341437 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e074092c-d612-4e28-a501-ca34eab4a14f-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:16.341490 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.341448 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e074092c-d612-4e28-a501-ca34eab4a14f-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:16.341490 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:16.341458 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpbmz\" (UniqueName: \"kubernetes.io/projected/e074092c-d612-4e28-a501-ca34eab4a14f-kube-api-access-kpbmz\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:35:17.078608 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:17.078577 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" Oct 31 15:35:17.078608 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:17.078590 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n" event={"ID":"e074092c-d612-4e28-a501-ca34eab4a14f","Type":"ContainerDied","Data":"ca2672bc1e362b3e56b90cd73c39991935b2375d3c75cb70b08f249e55475719"} Oct 31 15:35:17.079132 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:17.078650 4399 scope.go:122] "RemoveContainer" containerID="85717764815c836dd83d89c6c58adce858f3929e9d32f2daa78b3c11edd6a0fb" Oct 31 15:35:17.293852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:17.293818 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n"] Oct 31 15:35:17.343746 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:17.343686 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-d2c2n"] Oct 31 15:35:18.896895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:18.896856 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" path="/var/lib/kubelet/pods/e074092c-d612-4e28-a501-ca34eab4a14f/volumes" Oct 31 15:35:26.207639 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:35:26.207597 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:35:56.207180 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:35:56.207133 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:35:59.850248 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:59.850214 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb"] Oct 31 15:35:59.850657 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:59.850423 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerName="istio-proxy" containerID="cri-o://6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc" gracePeriod=30 Oct 31 15:35:59.860397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:59.860371 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7"] Oct 31 15:35:59.860651 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:59.860620 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" containerID="cri-o://a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2" gracePeriod=30 Oct 31 15:35:59.953942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:59.953795 4399 patch_prober.go:29] interesting pod/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:35:59.953942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:35:59.953881 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:36:05.136560 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.136534 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:36:05.161186 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.161165 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:36:05.235438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235414 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-token\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-token\") pod \"3fa09f46-6095-4714-93a4-5fd3047fd141\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " Oct 31 15:36:05.235583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235447 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-credential-socket\") pod \"3fa09f46-6095-4714-93a4-5fd3047fd141\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " Oct 31 15:36:05.235583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235484 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-certs\") pod \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " Oct 31 15:36:05.235583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235510 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-podinfo\") pod \"3fa09f46-6095-4714-93a4-5fd3047fd141\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " Oct 31 15:36:05.235583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235543 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-envoy\") pod \"3fa09f46-6095-4714-93a4-5fd3047fd141\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " Oct 31 15:36:05.235830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235587 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-kube-api-access-fpnk8\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-kube-api-access-fpnk8\") pod \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " Oct 31 15:36:05.235830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235617 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-data\") pod \"3fa09f46-6095-4714-93a4-5fd3047fd141\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " Oct 31 15:36:05.235830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235665 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-certs\") pod \"3fa09f46-6095-4714-93a4-5fd3047fd141\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " Oct 31 15:36:05.235830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235725 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-envoy\") pod \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " Oct 31 15:36:05.235830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235755 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-credential-socket" pod "3fa09f46-6095-4714-93a4-5fd3047fd141" (UID: "3fa09f46-6095-4714-93a4-5fd3047fd141"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.235830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235763 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-kube-api-access-974kq\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-kube-api-access-974kq\") pod \"3fa09f46-6095-4714-93a4-5fd3047fd141\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " Oct 31 15:36:05.235830 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235821 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-certs" pod "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" (UID: "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235834 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istiod-ca-cert\") pod \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235923 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-data\") pod \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235937 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-data" pod "3fa09f46-6095-4714-93a4-5fd3047fd141" (UID: "3fa09f46-6095-4714-93a4-5fd3047fd141"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235956 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-token\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-token\") pod \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.235989 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-socket\") pod \"3fa09f46-6095-4714-93a4-5fd3047fd141\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236043 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-podinfo\") pod \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236063 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-certs" pod "3fa09f46-6095-4714-93a4-5fd3047fd141" (UID: "3fa09f46-6095-4714-93a4-5fd3047fd141"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236096 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fa09f46-6095-4714-93a4-5fd3047fd141-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa09f46-6095-4714-93a4-5fd3047fd141-istiod-ca-cert\") pod \"3fa09f46-6095-4714-93a4-5fd3047fd141\" (UID: \"3fa09f46-6095-4714-93a4-5fd3047fd141\") " Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236158 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-socket\") pod \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " Oct 31 15:36:05.236319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236209 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-credential-socket\") pod \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\" (UID: \"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343\") " Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236430 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-data" pod "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" (UID: "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236798 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236782 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-socket" pod "3fa09f46-6095-4714-93a4-5fd3047fd141" (UID: "3fa09f46-6095-4714-93a4-5fd3047fd141"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236818 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236833 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236847 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236861 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236860 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-socket" pod "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" (UID: "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.236886 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istiod-ca-cert" pod "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" (UID: "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.237024 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-credential-socket" pod "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" (UID: "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.237903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.237318 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa09f46-6095-4714-93a4-5fd3047fd141-istiod-ca-cert" pod "3fa09f46-6095-4714-93a4-5fd3047fd141" (UID: "3fa09f46-6095-4714-93a4-5fd3047fd141"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:36:05.239779 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.238537 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-token" pod "3fa09f46-6095-4714-93a4-5fd3047fd141" (UID: "3fa09f46-6095-4714-93a4-5fd3047fd141"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:05.239779 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.238613 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-podinfo" pod "3fa09f46-6095-4714-93a4-5fd3047fd141" (UID: "3fa09f46-6095-4714-93a4-5fd3047fd141"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:36:05.239779 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.239026 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-kube-api-access-fpnk8" pod "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" (UID: "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343"). InnerVolumeSpecName "kube-api-access-fpnk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:05.239779 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.239060 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-envoy" pod "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" (UID: "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.239779 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.239438 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-podinfo" pod "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" (UID: "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:36:05.240104 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.240080 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-kube-api-access-974kq" pod "3fa09f46-6095-4714-93a4-5fd3047fd141" (UID: "3fa09f46-6095-4714-93a4-5fd3047fd141"). InnerVolumeSpecName "kube-api-access-974kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:05.240188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.240167 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-envoy" pod "3fa09f46-6095-4714-93a4-5fd3047fd141" (UID: "3fa09f46-6095-4714-93a4-5fd3047fd141"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:36:05.240448 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.240431 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-token" pod "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" (UID: "c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:36:05.338260 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338229 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338260 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338259 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338268 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338277 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338285 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpnk8\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-kube-api-access-fpnk8\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338295 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338304 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-974kq\" (UniqueName: \"kubernetes.io/projected/3fa09f46-6095-4714-93a4-5fd3047fd141-kube-api-access-974kq\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338313 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338321 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338329 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa09f46-6095-4714-93a4-5fd3047fd141-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338350 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338359 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa09f46-6095-4714-93a4-5fd3047fd141-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.338436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.338367 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:36:05.405892 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.405813 4399 generic.go:356] "Generic (PLEG): container finished" podID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerID="a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2" exitCode=0 Oct 31 15:36:05.405892 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.405858 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" Oct 31 15:36:05.406195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.405897 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" event={"ID":"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343","Type":"ContainerDied","Data":"a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2"} Oct 31 15:36:05.406195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.405960 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7" event={"ID":"c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343","Type":"ContainerDied","Data":"02bedae1c064d3564f868680b42f9ca3720b00d9a73ae9865073e9dcb120cca8"} Oct 31 15:36:05.406195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.405986 4399 scope.go:122] "RemoveContainer" containerID="a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2" Oct 31 15:36:05.407482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.407457 4399 generic.go:356] "Generic (PLEG): container finished" podID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerID="6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc" exitCode=0 Oct 31 15:36:05.407590 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.407499 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" event={"ID":"3fa09f46-6095-4714-93a4-5fd3047fd141","Type":"ContainerDied","Data":"6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc"} Oct 31 15:36:05.407590 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.407555 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" event={"ID":"3fa09f46-6095-4714-93a4-5fd3047fd141","Type":"ContainerDied","Data":"b9ce856fd285658e79e2b19573a1f54c1c616a22436bf15cd68c30fe964f624d"} Oct 31 15:36:05.407590 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.407507 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb" Oct 31 15:36:05.417412 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.417266 4399 scope.go:122] "RemoveContainer" containerID="a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2" Oct 31 15:36:05.417587 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:36:05.417551 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2\": container with ID starting with a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2 not found: ID does not exist" containerID="a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2" Oct 31 15:36:05.417670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.417596 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2"} err="failed to get container status \"a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2\": rpc error: code = NotFound desc = could not find container \"a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2\": container with ID starting with a6d43859a6924c8c3b9696e55184ee8b18df2319a951e7578ca50fbbecd8d2b2 not found: ID does not exist" Oct 31 15:36:05.417670 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.417614 4399 scope.go:122] "RemoveContainer" containerID="6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc" Oct 31 15:36:05.426412 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.426395 4399 scope.go:122] "RemoveContainer" containerID="6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc" Oct 31 15:36:05.426664 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:36:05.426646 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc\": container with ID starting with 6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc not found: ID does not exist" containerID="6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc" Oct 31 15:36:05.426751 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.426670 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc"} err="failed to get container status \"6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc\": rpc error: code = NotFound desc = could not find container \"6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc\": container with ID starting with 6f1268a48205b379132a60787eb7f0915f0af7b61852aa2e7702202ffbee21dc not found: ID does not exist" Oct 31 15:36:05.587612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.587581 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb"] Oct 31 15:36:05.591633 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.591608 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-5v5qb"] Oct 31 15:36:05.700372 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.700346 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7"] Oct 31 15:36:05.706899 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:05.706876 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-2ad95a-openshift-default-db9b776c-fzvb7"] Oct 31 15:36:06.897520 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:06.897489 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" path="/var/lib/kubelet/pods/3fa09f46-6095-4714-93a4-5fd3047fd141/volumes" Oct 31 15:36:06.898003 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:06.897985 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" path="/var/lib/kubelet/pods/c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343/volumes" Oct 31 15:36:26.206512 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:36:26.206478 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:36:37.322340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322303 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh"] Oct 31 15:36:37.322895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322880 4399 cpu_manager.go:414] "containerMap: removing container" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerName="istio-proxy" Oct 31 15:36:37.322949 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322897 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerName="istio-proxy" Oct 31 15:36:37.322949 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322911 4399 cpu_manager.go:414] "containerMap: removing container" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" Oct 31 15:36:37.322949 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322917 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" Oct 31 15:36:37.322949 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322935 4399 cpu_manager.go:414] "containerMap: removing container" podUID="a0e59334-0191-4990-b6e2-245dda9d2f3a" containerName="payload-pre-processing" Oct 31 15:36:37.322949 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322940 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0e59334-0191-4990-b6e2-245dda9d2f3a" containerName="payload-pre-processing" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322953 4399 cpu_manager.go:414] "containerMap: removing container" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322959 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322972 4399 cpu_manager.go:414] "containerMap: removing container" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322977 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322988 4399 cpu_manager.go:414] "containerMap: removing container" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.322992 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323004 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323009 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323016 4399 cpu_manager.go:414] "containerMap: removing container" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323020 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323027 4399 cpu_manager.go:414] "containerMap: removing container" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323031 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323039 4399 cpu_manager.go:414] "containerMap: removing container" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323043 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerName="istio-proxy" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323049 4399 cpu_manager.go:414] "containerMap: removing container" podUID="d3999cdd-5827-4754-bc49-e7e6917f4688" containerName="payload-processing" Oct 31 15:36:37.323113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323053 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3999cdd-5827-4754-bc49-e7e6917f4688" containerName="payload-processing" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323123 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0e59334-0191-4990-b6e2-245dda9d2f3a" containerName="payload-pre-processing" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323134 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="793b21ab-9740-45db-8700-b9d5d9809d44" containerName="istio-proxy" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323142 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="070c7696-f269-4704-949a-93c33f3aac79" containerName="istio-proxy" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323149 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd02a7e-dad7-4c8c-a279-93053f852bea" containerName="istio-proxy" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323154 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e074092c-d612-4e28-a501-ca34eab4a14f" containerName="istio-proxy" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323159 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fdf8f0-3dc7-4bb3-82a1-9c760f23d343" containerName="istio-proxy" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323164 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a67bace-573f-47d7-9094-9c926e0e5300" containerName="istio-proxy" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323169 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e8733f-c3e8-4521-8a37-eae42b12481a" containerName="istio-proxy" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323173 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3999cdd-5827-4754-bc49-e7e6917f4688" containerName="payload-processing" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323178 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fa09f46-6095-4714-93a4-5fd3047fd141" containerName="istio-proxy" Oct 31 15:36:37.323784 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.323182 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="4683bd12-f3a9-48a0-a120-c701c479d54c" containerName="istio-proxy" Oct 31 15:36:37.326619 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.326605 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.380595 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.380567 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c947d9-openshift-default-dockercfg-g7j9f\"" Oct 31 15:36:37.407774 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.407740 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.407938 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.407814 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-credential-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.407938 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.407928 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-certs\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.408059 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.408036 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istiod-ca-cert\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.408113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.408068 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-podinfo\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.408113 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.408093 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-data\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.408196 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.408171 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-envoy\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.408257 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.408217 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-token\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.408292 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.408255 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gq89\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-kube-api-access-6gq89\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.434684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.434653 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh"] Oct 31 15:36:37.509990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.509483 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istiod-ca-cert\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.509990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.509544 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-podinfo\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.509990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.509573 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-data\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.509990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.509606 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-envoy\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.509990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.509643 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-token\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.509990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.509689 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gq89\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-kube-api-access-6gq89\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.510602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.510286 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istiod-ca-cert\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.510602 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.510454 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-data\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.510976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.510949 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.511090 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.510989 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-credential-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.511683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.511662 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.511932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.511908 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-certs\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.512296 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.512273 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-certs\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.512556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.512538 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-credential-socket\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.514662 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.514638 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-envoy\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.515052 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.515026 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-podinfo\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.570727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.569978 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-token\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.570727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.569996 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gq89\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-kube-api-access-6gq89\") pod \"e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:37.640099 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:37.640022 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:38.073328 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:38.073303 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh"] Oct 31 15:36:38.075733 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:36:38.075661 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ab2c02_b50b_44c0_8073_44d25c2b40aa.slice/crio-74b39375c13aec41a99ce9a359be7ac49b81838981120259d67e7aa95ae4b996 WatchSource:0}: Error finding container 74b39375c13aec41a99ce9a359be7ac49b81838981120259d67e7aa95ae4b996: Status 404 returned error can't find the container with id 74b39375c13aec41a99ce9a359be7ac49b81838981120259d67e7aa95ae4b996 Oct 31 15:36:38.078362 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:38.078332 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:36:38.078430 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:38.078399 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:36:38.078430 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:38.078425 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:36:38.611959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:38.611923 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" event={"ID":"40ab2c02-b50b-44c0-8073-44d25c2b40aa","Type":"ContainerStarted","Data":"044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e"} Oct 31 15:36:38.612397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:38.612014 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" event={"ID":"40ab2c02-b50b-44c0-8073-44d25c2b40aa","Type":"ContainerStarted","Data":"74b39375c13aec41a99ce9a359be7ac49b81838981120259d67e7aa95ae4b996"} Oct 31 15:36:38.640375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:38.640345 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:38.732523 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:38.732462 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podStartSLOduration=1.732446744 podStartE2EDuration="1.732446744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:36:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:36:38.73064177 +0000 UTC m=+3041.941883181" watchObservedRunningTime="2026-10-31 15:36:38.732446744 +0000 UTC m=+3041.943688156" Oct 31 15:36:39.646304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:39.646277 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:40.623622 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:40.623591 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:40.624781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:36:40.624762 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:36:56.206231 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:36:56.206191 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:37:26.207913 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:37:26.207869 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:37:56.208162 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:37:56.208127 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:38:26.132055 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:26.129826 4399 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": read tcp 10.133.0.2:42844->10.133.0.137:15021: read: connection reset by peer" start-of-body= Oct 31 15:38:26.132055 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:26.129912 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": read tcp 10.133.0.2:42844->10.133.0.137:15021: read: connection reset by peer" Oct 31 15:38:26.206869 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:38:26.206835 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:38:26.259560 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:26.259526 4399 generic.go:356] "Generic (PLEG): container finished" podID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerID="044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e" exitCode=0 Oct 31 15:38:26.259777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:26.259598 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" event={"ID":"40ab2c02-b50b-44c0-8073-44d25c2b40aa","Type":"ContainerDied","Data":"044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e"} Oct 31 15:38:26.260079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:26.260056 4399 scope.go:122] "RemoveContainer" containerID="044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e" Oct 31 15:38:26.261729 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:26.261681 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:38:26.261792 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:26.261777 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:38:26.261829 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:26.261815 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:38:27.276630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:27.276594 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" event={"ID":"40ab2c02-b50b-44c0-8073-44d25c2b40aa","Type":"ContainerStarted","Data":"4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab"} Oct 31 15:38:27.640787 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:27.640671 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:38:27.640969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:27.640880 4399 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:38:27.640969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:27.640930 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 15:38:28.648722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:28.648379 4399 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:38:28.648722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:28.648446 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 15:38:29.649723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:29.640966 4399 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:38:29.649723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:29.641034 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 15:38:30.650724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:30.641117 4399 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:38:30.650724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:30.641182 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 15:38:31.640492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:31.640461 4399 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:38:31.640672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:31.640511 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 15:38:32.641329 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:32.641291 4399 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" start-of-body= Oct 31 15:38:32.641750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:32.641350 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.137:15021/healthz/ready\": dial tcp 10.133.0.137:15021: connect: connection refused" Oct 31 15:38:33.645593 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:33.645564 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:38:33.646030 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:33.646018 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:38:33.646804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:38:33.646784 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 15:38:56.207187 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:38:56.207145 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:39:26.207386 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:39:26.207348 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:39:56.207544 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:39:56.207505 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:40:26.207139 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:40:26.207105 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:40:32.796534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.796499 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29"] Oct 31 15:40:32.800365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.800347 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.813275 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.813232 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-metrics-certs\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.813395 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.813353 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aff3341-799d-4a08-a483-8b3f7478f941-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.813466 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.813424 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn5xj\" (UniqueName: \"kubernetes.io/projected/6aff3341-799d-4a08-a483-8b3f7478f941-kube-api-access-sn5xj\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.813533 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.813514 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.853196 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.853175 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c59395-serving-cert\"" Oct 31 15:40:32.905307 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.905282 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29"] Oct 31 15:40:32.914537 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.914512 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-metrics-certs\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.914683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.914557 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aff3341-799d-4a08-a483-8b3f7478f941-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.914683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.914593 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn5xj\" (UniqueName: \"kubernetes.io/projected/6aff3341-799d-4a08-a483-8b3f7478f941-kube-api-access-sn5xj\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.914683 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.914616 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.914904 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:40:32.914827 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-serving-cert: secret "maas-api-e2e-extmod-a-c59395-serving-cert" not found Oct 31 15:40:32.914963 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:40:32.914912 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls podName:6aff3341-799d-4a08-a483-8b3f7478f941 nodeName:}" failed. No retries permitted until 2026-10-31 15:40:33.414891782 +0000 UTC m=+3276.626133195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls") pod "maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" (UID: "6aff3341-799d-4a08-a483-8b3f7478f941") : secret "maas-api-e2e-extmod-a-c59395-serving-cert" not found Oct 31 15:40:32.915385 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.915366 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aff3341-799d-4a08-a483-8b3f7478f941-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.917676 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.917647 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-metrics-certs\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:32.971398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:32.971372 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn5xj\" (UniqueName: \"kubernetes.io/projected/6aff3341-799d-4a08-a483-8b3f7478f941-kube-api-access-sn5xj\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:33.421579 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:33.421546 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:33.424175 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:33.424144 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c59395-54884cb485-jgj29\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:33.714534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:33.714507 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:34.092122 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:34.092099 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29"] Oct 31 15:40:34.094657 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:40:34.094616 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aff3341_799d_4a08_a483_8b3f7478f941.slice/crio-72d1665fcad5f7c65e604de02e648a2eb40d81fd5229214fe9896e0aef01d647 WatchSource:0}: Error finding container 72d1665fcad5f7c65e604de02e648a2eb40d81fd5229214fe9896e0aef01d647: Status 404 returned error can't find the container with id 72d1665fcad5f7c65e604de02e648a2eb40d81fd5229214fe9896e0aef01d647 Oct 31 15:40:34.096318 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:34.096297 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 15:40:35.007975 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:35.007940 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" event={"ID":"6aff3341-799d-4a08-a483-8b3f7478f941","Type":"ContainerStarted","Data":"6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528"} Oct 31 15:40:35.008171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:35.007999 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" event={"ID":"6aff3341-799d-4a08-a483-8b3f7478f941","Type":"ContainerStarted","Data":"72d1665fcad5f7c65e604de02e648a2eb40d81fd5229214fe9896e0aef01d647"} Oct 31 15:40:35.008171 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:35.008098 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:35.160578 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:35.160472 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" podStartSLOduration=2.377553104 podStartE2EDuration="3.160458618s" totalImagesPullingTime="782.905514ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:40:35.15807927 +0000 UTC m=+3278.369320705" watchObservedRunningTime="2026-10-31 15:40:35.160458618 +0000 UTC m=+3278.371700030" Oct 31 15:40:41.020463 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:40:41.020375 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:40:56.207115 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:40:56.207080 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:41:26.206279 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:41:26.206240 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:41:47.023935 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.023901 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6"] Oct 31 15:41:47.028485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.028459 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:41:47.083596 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.083563 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6"] Oct 31 15:41:47.114177 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.114144 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d437b44-2517-4e0f-9870-a2c52a5c5839-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6\" (UID: \"4d437b44-2517-4e0f-9870-a2c52a5c5839\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:41:47.114344 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.114264 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx9zc\" (UniqueName: \"kubernetes.io/projected/4d437b44-2517-4e0f-9870-a2c52a5c5839-kube-api-access-xx9zc\") pod \"payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6\" (UID: \"4d437b44-2517-4e0f-9870-a2c52a5c5839\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:41:47.129108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.129083 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-69517a-dockercfg-g7cll\"" Oct 31 15:41:47.129108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.129083 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-69517a\"" Oct 31 15:41:47.216136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.216101 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d437b44-2517-4e0f-9870-a2c52a5c5839-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6\" (UID: \"4d437b44-2517-4e0f-9870-a2c52a5c5839\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:41:47.216293 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.216172 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx9zc\" (UniqueName: \"kubernetes.io/projected/4d437b44-2517-4e0f-9870-a2c52a5c5839-kube-api-access-xx9zc\") pod \"payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6\" (UID: \"4d437b44-2517-4e0f-9870-a2c52a5c5839\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:41:47.216938 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.216918 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d437b44-2517-4e0f-9870-a2c52a5c5839-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6\" (UID: \"4d437b44-2517-4e0f-9870-a2c52a5c5839\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:41:47.273054 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.273026 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx9zc\" (UniqueName: \"kubernetes.io/projected/4d437b44-2517-4e0f-9870-a2c52a5c5839-kube-api-access-xx9zc\") pod \"payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6\" (UID: \"4d437b44-2517-4e0f-9870-a2c52a5c5839\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:41:47.343516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.343453 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:41:47.725015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:47.724988 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6"] Oct 31 15:41:47.727373 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:41:47.727348 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d437b44_2517_4e0f_9870_a2c52a5c5839.slice/crio-fe5867bd6be28f0897cf6fec16f553fbc67fc2e4a7c26d3687250fe4253b38bf WatchSource:0}: Error finding container fe5867bd6be28f0897cf6fec16f553fbc67fc2e4a7c26d3687250fe4253b38bf: Status 404 returned error can't find the container with id fe5867bd6be28f0897cf6fec16f553fbc67fc2e4a7c26d3687250fe4253b38bf Oct 31 15:41:48.490909 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:48.490871 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" event={"ID":"4d437b44-2517-4e0f-9870-a2c52a5c5839","Type":"ContainerStarted","Data":"866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9"} Oct 31 15:41:48.491317 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:48.490934 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" event={"ID":"4d437b44-2517-4e0f-9870-a2c52a5c5839","Type":"ContainerStarted","Data":"fe5867bd6be28f0897cf6fec16f553fbc67fc2e4a7c26d3687250fe4253b38bf"} Oct 31 15:41:48.491317 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:48.490961 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:41:48.603845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:48.603788 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" podStartSLOduration=2.603771029 podStartE2EDuration="2.603771029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:41:48.602457635 +0000 UTC m=+3351.813699048" watchObservedRunningTime="2026-10-31 15:41:48.603771029 +0000 UTC m=+3351.815012638" Oct 31 15:41:56.206273 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:41:56.206233 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:41:59.500070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:41:59.500039 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:42:26.207532 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:42:26.207499 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:42:41.842679 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:41.842646 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx"] Oct 31 15:42:41.846755 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:41.846736 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:41.899061 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:41.899022 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-5a0cef-serving-cert\"" Oct 31 15:42:41.938427 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:41.938396 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgrl\" (UniqueName: \"kubernetes.io/projected/1d842564-cef1-48a8-b3c0-8b266a87375f-kube-api-access-rcgrl\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:41.938588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:41.938478 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-metrics-certs\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:41.938661 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:41.938600 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d842564-cef1-48a8-b3c0-8b266a87375f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:41.938756 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:41.938739 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:41.950904 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:41.950881 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx"] Oct 31 15:42:42.040232 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.040202 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d842564-cef1-48a8-b3c0-8b266a87375f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:42.040394 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.040275 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:42.040394 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.040381 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgrl\" (UniqueName: \"kubernetes.io/projected/1d842564-cef1-48a8-b3c0-8b266a87375f-kube-api-access-rcgrl\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:42.040476 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:42:42.040435 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-serving-cert: secret "maas-api-e2e-extmod-a-5a0cef-serving-cert" not found Oct 31 15:42:42.040476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.040439 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-metrics-certs\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:42.040557 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:42:42.040508 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls podName:1d842564-cef1-48a8-b3c0-8b266a87375f nodeName:}" failed. No retries permitted until 2026-10-31 15:42:42.540487085 +0000 UTC m=+3405.751728477 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls") pod "maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" (UID: "1d842564-cef1-48a8-b3c0-8b266a87375f") : secret "maas-api-e2e-extmod-a-5a0cef-serving-cert" not found Oct 31 15:42:42.040969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.040944 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d842564-cef1-48a8-b3c0-8b266a87375f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:42.042949 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.042929 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-metrics-certs\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:42.126482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.126402 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgrl\" (UniqueName: \"kubernetes.io/projected/1d842564-cef1-48a8-b3c0-8b266a87375f-kube-api-access-rcgrl\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:42.550344 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.550313 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:42.552934 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.552911 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:42.760023 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:42.759975 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:43.140518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:43.140430 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx"] Oct 31 15:42:43.149349 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:42:43.149323 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d842564_cef1_48a8_b3c0_8b266a87375f.slice/crio-2a56f178e0fdeee7a9e38d62117a56a56ead0f050fbe998932f41877b57e9834 WatchSource:0}: Error finding container 2a56f178e0fdeee7a9e38d62117a56a56ead0f050fbe998932f41877b57e9834: Status 404 returned error can't find the container with id 2a56f178e0fdeee7a9e38d62117a56a56ead0f050fbe998932f41877b57e9834 Oct 31 15:42:43.810396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:43.810361 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" event={"ID":"1d842564-cef1-48a8-b3c0-8b266a87375f","Type":"ContainerStarted","Data":"2a56f178e0fdeee7a9e38d62117a56a56ead0f050fbe998932f41877b57e9834"} Oct 31 15:42:44.819641 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:44.819608 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" event={"ID":"1d842564-cef1-48a8-b3c0-8b266a87375f","Type":"ContainerStarted","Data":"0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e"} Oct 31 15:42:44.820068 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:44.819751 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:44.935756 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:44.935680 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" podStartSLOduration=3.079760455 podStartE2EDuration="3.935662653s" totalImagesPullingTime="855.902198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:42:44.934188085 +0000 UTC m=+3408.145429498" watchObservedRunningTime="2026-10-31 15:42:44.935662653 +0000 UTC m=+3408.146904065" Oct 31 15:42:50.831863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:42:50.831832 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:42:56.209880 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:42:56.209840 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:43:26.208917 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:43:26.208881 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:43:37.787357 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:37.787279 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29"] Oct 31 15:43:37.787775 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:37.787474 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" podUID="6aff3341-799d-4a08-a483-8b3f7478f941" containerName="maas-api" containerID="cri-o://6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528" gracePeriod=30 Oct 31 15:43:38.080424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.080387 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:43:38.126433 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.126400 4399 generic.go:356] "Generic (PLEG): container finished" podID="6aff3341-799d-4a08-a483-8b3f7478f941" containerID="6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528" exitCode=0 Oct 31 15:43:38.126591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.126456 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" Oct 31 15:43:38.126591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.126479 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" event={"ID":"6aff3341-799d-4a08-a483-8b3f7478f941","Type":"ContainerDied","Data":"6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528"} Oct 31 15:43:38.126591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.126535 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29" event={"ID":"6aff3341-799d-4a08-a483-8b3f7478f941","Type":"ContainerDied","Data":"72d1665fcad5f7c65e604de02e648a2eb40d81fd5229214fe9896e0aef01d647"} Oct 31 15:43:38.126591 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.126560 4399 scope.go:122] "RemoveContainer" containerID="6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528" Oct 31 15:43:38.136402 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.136383 4399 scope.go:122] "RemoveContainer" containerID="6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528" Oct 31 15:43:38.136751 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:43:38.136731 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528\": container with ID starting with 6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528 not found: ID does not exist" containerID="6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528" Oct 31 15:43:38.136841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.136761 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528"} err="failed to get container status \"6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528\": rpc error: code = NotFound desc = could not find container \"6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528\": container with ID starting with 6376dfc4554bfa0f68f9b5fddf24b62006d38509efbc47da9e4d783193a1e528 not found: ID does not exist" Oct 31 15:43:38.191498 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.191458 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aff3341-799d-4a08-a483-8b3f7478f941-kube-api-access-sn5xj\" (UniqueName: \"kubernetes.io/projected/6aff3341-799d-4a08-a483-8b3f7478f941-kube-api-access-sn5xj\") pod \"6aff3341-799d-4a08-a483-8b3f7478f941\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " Oct 31 15:43:38.191765 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.191744 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls\") pod \"6aff3341-799d-4a08-a483-8b3f7478f941\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " Oct 31 15:43:38.191882 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.191812 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-metrics-certs\") pod \"6aff3341-799d-4a08-a483-8b3f7478f941\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " Oct 31 15:43:38.191882 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.191850 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aff3341-799d-4a08-a483-8b3f7478f941-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aff3341-799d-4a08-a483-8b3f7478f941-openshift-service-ca\") pod \"6aff3341-799d-4a08-a483-8b3f7478f941\" (UID: \"6aff3341-799d-4a08-a483-8b3f7478f941\") " Oct 31 15:43:38.192588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.192564 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aff3341-799d-4a08-a483-8b3f7478f941-openshift-service-ca" pod "6aff3341-799d-4a08-a483-8b3f7478f941" (UID: "6aff3341-799d-4a08-a483-8b3f7478f941"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:43:38.194301 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.194260 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aff3341-799d-4a08-a483-8b3f7478f941-kube-api-access-sn5xj" pod "6aff3341-799d-4a08-a483-8b3f7478f941" (UID: "6aff3341-799d-4a08-a483-8b3f7478f941"). InnerVolumeSpecName "kube-api-access-sn5xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:43:38.194654 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.194619 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls" pod "6aff3341-799d-4a08-a483-8b3f7478f941" (UID: "6aff3341-799d-4a08-a483-8b3f7478f941"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:43:38.201531 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.201497 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-metrics-certs" pod "6aff3341-799d-4a08-a483-8b3f7478f941" (UID: "6aff3341-799d-4a08-a483-8b3f7478f941"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:43:38.293919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.293891 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:38.293919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.293915 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6aff3341-799d-4a08-a483-8b3f7478f941-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:38.294081 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.293927 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6aff3341-799d-4a08-a483-8b3f7478f941-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:38.294081 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.293935 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn5xj\" (UniqueName: \"kubernetes.io/projected/6aff3341-799d-4a08-a483-8b3f7478f941-kube-api-access-sn5xj\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:43:38.547246 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.547207 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29"] Oct 31 15:43:38.550448 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.550423 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c59395-54884cb485-jgj29"] Oct 31 15:43:38.896259 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:43:38.896225 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aff3341-799d-4a08-a483-8b3f7478f941" path="/var/lib/kubelet/pods/6aff3341-799d-4a08-a483-8b3f7478f941/volumes" Oct 31 15:43:56.207156 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:43:56.207121 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:44:26.207779 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:44:26.207745 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:44:56.208952 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:44:56.208910 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:45:00.223151 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.223107 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq"] Oct 31 15:45:00.223832 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.223816 4399 cpu_manager.go:414] "containerMap: removing container" podUID="6aff3341-799d-4a08-a483-8b3f7478f941" containerName="maas-api" Oct 31 15:45:00.223894 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.223834 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aff3341-799d-4a08-a483-8b3f7478f941" containerName="maas-api" Oct 31 15:45:00.223941 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.223931 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aff3341-799d-4a08-a483-8b3f7478f941" containerName="maas-api" Oct 31 15:45:00.227418 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.227402 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:00.238413 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.238380 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h"] Oct 31 15:45:00.242894 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.242875 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:00.260660 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.260638 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq\" (UID: \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:00.260768 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.260722 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlmkf\" (UniqueName: \"kubernetes.io/projected/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-kube-api-access-zlmkf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq\" (UID: \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:00.334640 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.334609 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 15:45:00.362114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.362076 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq\" (UID: \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:00.362345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.362137 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4ppk\" (UniqueName: \"kubernetes.io/projected/8e062f91-e1dc-4644-830f-64054e2be83a-kube-api-access-f4ppk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h\" (UID: \"8e062f91-e1dc-4644-830f-64054e2be83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:00.362345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.362182 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e062f91-e1dc-4644-830f-64054e2be83a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h\" (UID: \"8e062f91-e1dc-4644-830f-64054e2be83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:00.362345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.362256 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlmkf\" (UniqueName: \"kubernetes.io/projected/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-kube-api-access-zlmkf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq\" (UID: \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:00.362928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.362894 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq\" (UID: \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:00.396593 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.396554 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h"] Oct 31 15:45:00.419194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.419156 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlmkf\" (UniqueName: \"kubernetes.io/projected/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-kube-api-access-zlmkf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq\" (UID: \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:00.427853 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.427803 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq"] Oct 31 15:45:00.463095 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.463066 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4ppk\" (UniqueName: \"kubernetes.io/projected/8e062f91-e1dc-4644-830f-64054e2be83a-kube-api-access-f4ppk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h\" (UID: \"8e062f91-e1dc-4644-830f-64054e2be83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:00.463304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.463117 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e062f91-e1dc-4644-830f-64054e2be83a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h\" (UID: \"8e062f91-e1dc-4644-830f-64054e2be83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:00.463852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.463825 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e062f91-e1dc-4644-830f-64054e2be83a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h\" (UID: \"8e062f91-e1dc-4644-830f-64054e2be83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:00.521161 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.521087 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4ppk\" (UniqueName: \"kubernetes.io/projected/8e062f91-e1dc-4644-830f-64054e2be83a-kube-api-access-f4ppk\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h\" (UID: \"8e062f91-e1dc-4644-830f-64054e2be83a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:00.541903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.541869 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:00.556915 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.556890 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:00.935228 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.935154 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq"] Oct 31 15:45:00.977547 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:45:00.977517 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e9cca1_362a_4150_af1f_4d36bb1e9e2d.slice/crio-04f559507bc2f9bf62c082be9f94b6398217486120532984aae46a90766dc698 WatchSource:0}: Error finding container 04f559507bc2f9bf62c082be9f94b6398217486120532984aae46a90766dc698: Status 404 returned error can't find the container with id 04f559507bc2f9bf62c082be9f94b6398217486120532984aae46a90766dc698 Oct 31 15:45:00.988234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:00.988206 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h"] Oct 31 15:45:00.990549 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:45:00.990519 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e062f91_e1dc_4644_830f_64054e2be83a.slice/crio-e5246e9f356488c73af3d154a6a5462efe779c693d008606246ac6519bad8c36 WatchSource:0}: Error finding container e5246e9f356488c73af3d154a6a5462efe779c693d008606246ac6519bad8c36: Status 404 returned error can't find the container with id e5246e9f356488c73af3d154a6a5462efe779c693d008606246ac6519bad8c36 Oct 31 15:45:01.697147 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:01.697111 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" event={"ID":"8e062f91-e1dc-4644-830f-64054e2be83a","Type":"ContainerStarted","Data":"eb209bed766e466484bcede9575f12e8642eb686fda8f68c714742b2c921b440"} Oct 31 15:45:01.697618 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:01.697173 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" event={"ID":"8e062f91-e1dc-4644-830f-64054e2be83a","Type":"ContainerStarted","Data":"e5246e9f356488c73af3d154a6a5462efe779c693d008606246ac6519bad8c36"} Oct 31 15:45:01.699078 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:01.699043 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" event={"ID":"73e9cca1-362a-4150-af1f-4d36bb1e9e2d","Type":"ContainerStarted","Data":"f2a7dc84877dd71dd416f0d3920a048455ae1402950e0092fbbddfe3a8ac6327"} Oct 31 15:45:01.699221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:01.699119 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" event={"ID":"73e9cca1-362a-4150-af1f-4d36bb1e9e2d","Type":"ContainerStarted","Data":"04f559507bc2f9bf62c082be9f94b6398217486120532984aae46a90766dc698"} Oct 31 15:45:01.814007 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:01.813943 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" podStartSLOduration=1.8139234800000001 podStartE2EDuration="1.81392348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:01.812235886 +0000 UTC m=+3545.023477334" watchObservedRunningTime="2026-10-31 15:45:01.81392348 +0000 UTC m=+3545.025164893" Oct 31 15:45:01.925280 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:01.925222 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" podStartSLOduration=1.9252077920000001 podStartE2EDuration="1.925207792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:01.923822839 +0000 UTC m=+3545.135064252" watchObservedRunningTime="2026-10-31 15:45:01.925207792 +0000 UTC m=+3545.136449244" Oct 31 15:45:02.706963 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:02.706919 4399 generic.go:356] "Generic (PLEG): container finished" podID="8e062f91-e1dc-4644-830f-64054e2be83a" containerID="eb209bed766e466484bcede9575f12e8642eb686fda8f68c714742b2c921b440" exitCode=60 Oct 31 15:45:02.707438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:02.707020 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" event={"ID":"8e062f91-e1dc-4644-830f-64054e2be83a","Type":"ContainerDied","Data":"eb209bed766e466484bcede9575f12e8642eb686fda8f68c714742b2c921b440"} Oct 31 15:45:02.707438 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:02.707271 4399 scope.go:122] "RemoveContainer" containerID="eb209bed766e466484bcede9575f12e8642eb686fda8f68c714742b2c921b440" Oct 31 15:45:02.709665 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:02.709646 4399 generic.go:356] "Generic (PLEG): container finished" podID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerID="f2a7dc84877dd71dd416f0d3920a048455ae1402950e0092fbbddfe3a8ac6327" exitCode=60 Oct 31 15:45:02.709806 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:02.709742 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" event={"ID":"73e9cca1-362a-4150-af1f-4d36bb1e9e2d","Type":"ContainerDied","Data":"f2a7dc84877dd71dd416f0d3920a048455ae1402950e0092fbbddfe3a8ac6327"} Oct 31 15:45:02.710018 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:02.710002 4399 scope.go:122] "RemoveContainer" containerID="f2a7dc84877dd71dd416f0d3920a048455ae1402950e0092fbbddfe3a8ac6327" Oct 31 15:45:03.720666 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:03.720629 4399 generic.go:356] "Generic (PLEG): container finished" podID="8e062f91-e1dc-4644-830f-64054e2be83a" containerID="f6b8a9af6d8126ce9fe1db8cb33339d14ac6e832558710f0dd0777c697950353" exitCode=60 Oct 31 15:45:03.721157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:03.720737 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" event={"ID":"8e062f91-e1dc-4644-830f-64054e2be83a","Type":"ContainerDied","Data":"f6b8a9af6d8126ce9fe1db8cb33339d14ac6e832558710f0dd0777c697950353"} Oct 31 15:45:03.721157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:03.720807 4399 scope.go:122] "RemoveContainer" containerID="eb209bed766e466484bcede9575f12e8642eb686fda8f68c714742b2c921b440" Oct 31 15:45:03.721157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:03.721072 4399 scope.go:122] "RemoveContainer" containerID="f6b8a9af6d8126ce9fe1db8cb33339d14ac6e832558710f0dd0777c697950353" Oct 31 15:45:03.721398 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:45:03.721376 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h_odh-ai-gateway-infra(8e062f91-e1dc-4644-830f-64054e2be83a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" Oct 31 15:45:03.723145 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:03.723120 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" event={"ID":"73e9cca1-362a-4150-af1f-4d36bb1e9e2d","Type":"ContainerStarted","Data":"64d4a81c20898ca2be40df99beb6fac65b87184d29e34a335c61b5426f6db86b"} Oct 31 15:45:04.731021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:04.730979 4399 scope.go:122] "RemoveContainer" containerID="f6b8a9af6d8126ce9fe1db8cb33339d14ac6e832558710f0dd0777c697950353" Oct 31 15:45:04.731505 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:45:04.731313 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h_odh-ai-gateway-infra(8e062f91-e1dc-4644-830f-64054e2be83a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" Oct 31 15:45:04.732688 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:04.732665 4399 generic.go:356] "Generic (PLEG): container finished" podID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerID="64d4a81c20898ca2be40df99beb6fac65b87184d29e34a335c61b5426f6db86b" exitCode=60 Oct 31 15:45:04.732850 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:04.732746 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" event={"ID":"73e9cca1-362a-4150-af1f-4d36bb1e9e2d","Type":"ContainerDied","Data":"64d4a81c20898ca2be40df99beb6fac65b87184d29e34a335c61b5426f6db86b"} Oct 31 15:45:04.732850 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:04.732817 4399 scope.go:122] "RemoveContainer" containerID="f2a7dc84877dd71dd416f0d3920a048455ae1402950e0092fbbddfe3a8ac6327" Oct 31 15:45:04.733034 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:04.733019 4399 scope.go:122] "RemoveContainer" containerID="64d4a81c20898ca2be40df99beb6fac65b87184d29e34a335c61b5426f6db86b" Oct 31 15:45:04.733320 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:45:04.733294 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq_odh-ai-gateway-infra(73e9cca1-362a-4150-af1f-4d36bb1e9e2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" Oct 31 15:45:05.740062 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:05.740022 4399 scope.go:122] "RemoveContainer" containerID="64d4a81c20898ca2be40df99beb6fac65b87184d29e34a335c61b5426f6db86b" Oct 31 15:45:05.740460 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:45:05.740269 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq_odh-ai-gateway-infra(73e9cca1-362a-4150-af1f-4d36bb1e9e2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" Oct 31 15:45:10.837744 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.837654 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg"] Oct 31 15:45:10.843024 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.843005 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:10.875969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.875934 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b506a697-275d-45ee-8922-521497e1880c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e34efa-gvtqg\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:10.876077 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.875990 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twf5c\" (UniqueName: \"kubernetes.io/projected/b506a697-275d-45ee-8922-521497e1880c-kube-api-access-twf5c\") pod \"maas-api-revoke-sub-f1e34efa-gvtqg\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:10.876172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.876152 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b506a697-275d-45ee-8922-521497e1880c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e34efa-gvtqg\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:10.898878 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.898854 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg"] Oct 31 15:45:10.966684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.966651 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 15:45:10.976934 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.976905 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b506a697-275d-45ee-8922-521497e1880c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e34efa-gvtqg\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:10.977072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.976958 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twf5c\" (UniqueName: \"kubernetes.io/projected/b506a697-275d-45ee-8922-521497e1880c-kube-api-access-twf5c\") pod \"maas-api-revoke-sub-f1e34efa-gvtqg\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:10.977175 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.977161 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b506a697-275d-45ee-8922-521497e1880c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e34efa-gvtqg\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:10.977690 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.977664 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b506a697-275d-45ee-8922-521497e1880c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1e34efa-gvtqg\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:10.980086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:10.980064 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b506a697-275d-45ee-8922-521497e1880c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1e34efa-gvtqg\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:11.033279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.033246 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twf5c\" (UniqueName: \"kubernetes.io/projected/b506a697-275d-45ee-8922-521497e1880c-kube-api-access-twf5c\") pod \"maas-api-revoke-sub-f1e34efa-gvtqg\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:11.156396 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.156100 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:11.546817 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.546786 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg"] Oct 31 15:45:11.548050 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:45:11.548012 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb506a697_275d_45ee_8922_521497e1880c.slice/crio-4e31ff52c3197d2181b0c658defe23d168c1b3cbaf67e5345361af19ba6fe9e3 WatchSource:0}: Error finding container 4e31ff52c3197d2181b0c658defe23d168c1b3cbaf67e5345361af19ba6fe9e3: Status 404 returned error can't find the container with id 4e31ff52c3197d2181b0c658defe23d168c1b3cbaf67e5345361af19ba6fe9e3 Oct 31 15:45:11.776815 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.776783 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" event={"ID":"b506a697-275d-45ee-8922-521497e1880c","Type":"ContainerStarted","Data":"4e31ff52c3197d2181b0c658defe23d168c1b3cbaf67e5345361af19ba6fe9e3"} Oct 31 15:45:11.871561 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.871533 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv"] Oct 31 15:45:11.875559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.875539 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.888162 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.888136 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db214e1-a5bb-414f-b55a-23b0130f9115-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.888256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.888199 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87x8b\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-kube-api-access-87x8b\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.888317 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.888283 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.888364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.888352 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.888409 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.888398 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.888456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.888437 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.888540 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.888521 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.888592 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.888560 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.888751 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.888729 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.934812 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.934778 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv"] Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.990251 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.990298 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.990365 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.990468 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db214e1-a5bb-414f-b55a-23b0130f9115-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.990544 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87x8b\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-kube-api-access-87x8b\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.990593 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.990630 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.990659 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.990726 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.991138 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.991223 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.991448 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.991503 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.991497 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db214e1-a5bb-414f-b55a-23b0130f9115-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.993181 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.991595 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.995025 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.994986 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:11.995366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:11.995338 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:12.047840 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.047817 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:12.048107 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.048089 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87x8b\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-kube-api-access-87x8b\") pod \"maas-default-gateway-openshift-default-67bf986f99-wlhfv\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:12.189777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.189682 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:12.622204 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.622179 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv"] Oct 31 15:45:12.623479 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:45:12.623456 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db214e1_a5bb_414f_b55a_23b0130f9115.slice/crio-bd5fb6240ec921dc3abd4ab1c32c04214eaafe9231c56b282d68471f1d9ff3a3 WatchSource:0}: Error finding container bd5fb6240ec921dc3abd4ab1c32c04214eaafe9231c56b282d68471f1d9ff3a3: Status 404 returned error can't find the container with id bd5fb6240ec921dc3abd4ab1c32c04214eaafe9231c56b282d68471f1d9ff3a3 Oct 31 15:45:12.626149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.626120 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:12.626220 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.626194 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:12.626266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.626225 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:12.785989 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.785899 4399 generic.go:356] "Generic (PLEG): container finished" podID="b506a697-275d-45ee-8922-521497e1880c" containerID="5a770388fe668cc73dd227c2b5d0e15f273b30f71d39acb7e3b676e729de8e40" exitCode=0 Oct 31 15:45:12.786163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.785986 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" event={"ID":"b506a697-275d-45ee-8922-521497e1880c","Type":"ContainerDied","Data":"5a770388fe668cc73dd227c2b5d0e15f273b30f71d39acb7e3b676e729de8e40"} Oct 31 15:45:12.787886 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.787857 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" event={"ID":"9db214e1-a5bb-414f-b55a-23b0130f9115","Type":"ContainerStarted","Data":"cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc"} Oct 31 15:45:12.788048 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:12.787902 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" event={"ID":"9db214e1-a5bb-414f-b55a-23b0130f9115","Type":"ContainerStarted","Data":"bd5fb6240ec921dc3abd4ab1c32c04214eaafe9231c56b282d68471f1d9ff3a3"} Oct 31 15:45:13.019008 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:13.018955 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podStartSLOduration=2.018940468 podStartE2EDuration="2.018940468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:13.011150466 +0000 UTC m=+3556.222391883" watchObservedRunningTime="2026-10-31 15:45:13.018940468 +0000 UTC m=+3556.230181879" Oct 31 15:45:13.190766 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:13.190658 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:13.192168 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:13.192144 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wlhfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 15:45:13.192305 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:13.192200 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 15:45:13.988151 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:13.988128 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:14.016954 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.016931 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b506a697-275d-45ee-8922-521497e1880c-kube-api-access-twf5c\" (UniqueName: \"kubernetes.io/projected/b506a697-275d-45ee-8922-521497e1880c-kube-api-access-twf5c\") pod \"b506a697-275d-45ee-8922-521497e1880c\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " Oct 31 15:45:14.017095 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.017043 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b506a697-275d-45ee-8922-521497e1880c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b506a697-275d-45ee-8922-521497e1880c-maas-api-openshift-ca\") pod \"b506a697-275d-45ee-8922-521497e1880c\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " Oct 31 15:45:14.017166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.017100 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b506a697-275d-45ee-8922-521497e1880c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b506a697-275d-45ee-8922-521497e1880c-maas-api-xks-ca\") pod \"b506a697-275d-45ee-8922-521497e1880c\" (UID: \"b506a697-275d-45ee-8922-521497e1880c\") " Oct 31 15:45:14.017370 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.017342 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b506a697-275d-45ee-8922-521497e1880c-maas-api-openshift-ca" pod "b506a697-275d-45ee-8922-521497e1880c" (UID: "b506a697-275d-45ee-8922-521497e1880c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:14.017453 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.017423 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b506a697-275d-45ee-8922-521497e1880c-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:14.019236 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.019213 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b506a697-275d-45ee-8922-521497e1880c-maas-api-xks-ca" pod "b506a697-275d-45ee-8922-521497e1880c" (UID: "b506a697-275d-45ee-8922-521497e1880c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:45:14.019522 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.019271 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b506a697-275d-45ee-8922-521497e1880c-kube-api-access-twf5c" pod "b506a697-275d-45ee-8922-521497e1880c" (UID: "b506a697-275d-45ee-8922-521497e1880c"). InnerVolumeSpecName "kube-api-access-twf5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:14.118334 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.118253 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b506a697-275d-45ee-8922-521497e1880c-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:14.118334 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.118281 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twf5c\" (UniqueName: \"kubernetes.io/projected/b506a697-275d-45ee-8922-521497e1880c-kube-api-access-twf5c\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:14.190194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.190159 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wlhfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 15:45:14.190366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.190221 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 15:45:14.800765 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.800728 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" event={"ID":"b506a697-275d-45ee-8922-521497e1880c","Type":"ContainerDied","Data":"4e31ff52c3197d2181b0c658defe23d168c1b3cbaf67e5345361af19ba6fe9e3"} Oct 31 15:45:14.800929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.800786 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e31ff52c3197d2181b0c658defe23d168c1b3cbaf67e5345361af19ba6fe9e3" Oct 31 15:45:14.800929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:14.800748 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg" Oct 31 15:45:15.193781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.191440 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wlhfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 15:45:15.193781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.191504 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 15:45:15.638328 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.638277 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f"] Oct 31 15:45:15.639464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.639438 4399 cpu_manager.go:414] "containerMap: removing container" podUID="b506a697-275d-45ee-8922-521497e1880c" containerName="revoke-keys" Oct 31 15:45:15.639464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.639459 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b506a697-275d-45ee-8922-521497e1880c" containerName="revoke-keys" Oct 31 15:45:15.639689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.639613 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="b506a697-275d-45ee-8922-521497e1880c" containerName="revoke-keys" Oct 31 15:45:15.665723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.662148 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:15.718797 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.718590 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b18df8-dockercfg-x27xp\"" Oct 31 15:45:15.719685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.719658 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b18df8\"" Oct 31 15:45:15.723160 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.720765 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f"] Oct 31 15:45:15.799103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.799066 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb0f96e-61db-416c-8486-b2d61faac230-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f\" (UID: \"0bb0f96e-61db-416c-8486-b2d61faac230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:15.799304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.799137 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljq2p\" (UniqueName: \"kubernetes.io/projected/0bb0f96e-61db-416c-8486-b2d61faac230-kube-api-access-ljq2p\") pod \"payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f\" (UID: \"0bb0f96e-61db-416c-8486-b2d61faac230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:15.818964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.818935 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh"] Oct 31 15:45:15.823142 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.823126 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:15.879660 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.879624 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh"] Oct 31 15:45:15.900721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.900625 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb0f96e-61db-416c-8486-b2d61faac230-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f\" (UID: \"0bb0f96e-61db-416c-8486-b2d61faac230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:15.900879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.900689 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljq2p\" (UniqueName: \"kubernetes.io/projected/0bb0f96e-61db-416c-8486-b2d61faac230-kube-api-access-ljq2p\") pod \"payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f\" (UID: \"0bb0f96e-61db-416c-8486-b2d61faac230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:15.900879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.900825 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e7eb52-dc71-45e9-8938-d1eefc8f0526-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh\" (UID: \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:15.901066 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.901041 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pxrn\" (UniqueName: \"kubernetes.io/projected/30e7eb52-dc71-45e9-8938-d1eefc8f0526-kube-api-access-9pxrn\") pod \"payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh\" (UID: \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:15.901608 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.901586 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb0f96e-61db-416c-8486-b2d61faac230-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f\" (UID: \"0bb0f96e-61db-416c-8486-b2d61faac230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:15.968815 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.968777 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljq2p\" (UniqueName: \"kubernetes.io/projected/0bb0f96e-61db-416c-8486-b2d61faac230-kube-api-access-ljq2p\") pod \"payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f\" (UID: \"0bb0f96e-61db-416c-8486-b2d61faac230\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:15.997687 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:15.997658 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:16.002427 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.002401 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pxrn\" (UniqueName: \"kubernetes.io/projected/30e7eb52-dc71-45e9-8938-d1eefc8f0526-kube-api-access-9pxrn\") pod \"payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh\" (UID: \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:16.002516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.002503 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e7eb52-dc71-45e9-8938-d1eefc8f0526-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh\" (UID: \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:16.003229 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.003209 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e7eb52-dc71-45e9-8938-d1eefc8f0526-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh\" (UID: \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:16.066801 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.066769 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pxrn\" (UniqueName: \"kubernetes.io/projected/30e7eb52-dc71-45e9-8938-d1eefc8f0526-kube-api-access-9pxrn\") pod \"payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh\" (UID: \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:16.136945 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.136914 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:16.190369 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.190338 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wlhfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 15:45:16.190518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.190387 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 15:45:16.401655 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.401627 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f"] Oct 31 15:45:16.402991 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:45:16.402963 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bb0f96e_61db_416c_8486_b2d61faac230.slice/crio-f66ae51b69043bd5553130f7939dceb2d3bbac058b6a4a52b9f851e1c05d22d1 WatchSource:0}: Error finding container f66ae51b69043bd5553130f7939dceb2d3bbac058b6a4a52b9f851e1c05d22d1: Status 404 returned error can't find the container with id f66ae51b69043bd5553130f7939dceb2d3bbac058b6a4a52b9f851e1c05d22d1 Oct 31 15:45:16.568086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.566627 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh"] Oct 31 15:45:16.568173 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:45:16.568106 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30e7eb52_dc71_45e9_8938_d1eefc8f0526.slice/crio-2e3b60a8dbe440e6f42a394dd554632ac92d208cc3e55b2254346d326ea374de WatchSource:0}: Error finding container 2e3b60a8dbe440e6f42a394dd554632ac92d208cc3e55b2254346d326ea374de: Status 404 returned error can't find the container with id 2e3b60a8dbe440e6f42a394dd554632ac92d208cc3e55b2254346d326ea374de Oct 31 15:45:16.815765 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.815731 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" event={"ID":"30e7eb52-dc71-45e9-8938-d1eefc8f0526","Type":"ContainerStarted","Data":"b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9"} Oct 31 15:45:16.815965 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.815794 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" event={"ID":"30e7eb52-dc71-45e9-8938-d1eefc8f0526","Type":"ContainerStarted","Data":"2e3b60a8dbe440e6f42a394dd554632ac92d208cc3e55b2254346d326ea374de"} Oct 31 15:45:16.815965 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.815951 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:16.817461 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.817437 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" event={"ID":"0bb0f96e-61db-416c-8486-b2d61faac230","Type":"ContainerStarted","Data":"a69c6b17dba4db1a89d2673cea38a5b91737343cd43e2305ef662d075840ff65"} Oct 31 15:45:16.817584 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.817471 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" event={"ID":"0bb0f96e-61db-416c-8486-b2d61faac230","Type":"ContainerStarted","Data":"f66ae51b69043bd5553130f7939dceb2d3bbac058b6a4a52b9f851e1c05d22d1"} Oct 31 15:45:16.817584 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.817490 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:16.889869 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.889831 4399 scope.go:122] "RemoveContainer" containerID="f6b8a9af6d8126ce9fe1db8cb33339d14ac6e832558710f0dd0777c697950353" Oct 31 15:45:16.928230 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:16.928178 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" podStartSLOduration=1.928165833 podStartE2EDuration="1.928165833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:16.926684667 +0000 UTC m=+3560.137926082" watchObservedRunningTime="2026-10-31 15:45:16.928165833 +0000 UTC m=+3560.139407246" Oct 31 15:45:17.069157 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:17.069032 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" podStartSLOduration=2.068997081 podStartE2EDuration="2.068997081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:17.048218683 +0000 UTC m=+3560.259460096" watchObservedRunningTime="2026-10-31 15:45:17.068997081 +0000 UTC m=+3560.280238494" Oct 31 15:45:17.190804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:17.190173 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wlhfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 15:45:17.190804 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:17.190307 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 15:45:17.824992 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:17.824955 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" event={"ID":"8e062f91-e1dc-4644-830f-64054e2be83a","Type":"ContainerStarted","Data":"969c51d9fb8f88dc7c65a81f6faeb0cf6278894a03e1cfa1abaf0ceb2efe63a3"} Oct 31 15:45:17.890074 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:17.890048 4399 scope.go:122] "RemoveContainer" containerID="64d4a81c20898ca2be40df99beb6fac65b87184d29e34a335c61b5426f6db86b" Oct 31 15:45:18.190546 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:18.190512 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wlhfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 15:45:18.190726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:18.190561 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 15:45:18.833556 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:18.833520 4399 generic.go:356] "Generic (PLEG): container finished" podID="8e062f91-e1dc-4644-830f-64054e2be83a" containerID="969c51d9fb8f88dc7c65a81f6faeb0cf6278894a03e1cfa1abaf0ceb2efe63a3" exitCode=60 Oct 31 15:45:18.834018 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:18.833578 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" event={"ID":"8e062f91-e1dc-4644-830f-64054e2be83a","Type":"ContainerDied","Data":"969c51d9fb8f88dc7c65a81f6faeb0cf6278894a03e1cfa1abaf0ceb2efe63a3"} Oct 31 15:45:18.834018 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:18.833717 4399 scope.go:122] "RemoveContainer" containerID="f6b8a9af6d8126ce9fe1db8cb33339d14ac6e832558710f0dd0777c697950353" Oct 31 15:45:18.834170 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:18.834129 4399 scope.go:122] "RemoveContainer" containerID="969c51d9fb8f88dc7c65a81f6faeb0cf6278894a03e1cfa1abaf0ceb2efe63a3" Oct 31 15:45:18.834512 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:45:18.834478 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h_odh-ai-gateway-infra(8e062f91-e1dc-4644-830f-64054e2be83a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" Oct 31 15:45:18.836146 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:18.836121 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" event={"ID":"73e9cca1-362a-4150-af1f-4d36bb1e9e2d","Type":"ContainerStarted","Data":"4495383f894e4049457cc85460f04e98f2304610fa4a8c71193c71ab6260be50"} Oct 31 15:45:18.948423 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:18.948388 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h"] Oct 31 15:45:19.190956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:19.190870 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wlhfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 15:45:19.190956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:19.190930 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 15:45:19.850182 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:19.850146 4399 generic.go:356] "Generic (PLEG): container finished" podID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerID="4495383f894e4049457cc85460f04e98f2304610fa4a8c71193c71ab6260be50" exitCode=60 Oct 31 15:45:19.850611 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:19.850216 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" event={"ID":"73e9cca1-362a-4150-af1f-4d36bb1e9e2d","Type":"ContainerDied","Data":"4495383f894e4049457cc85460f04e98f2304610fa4a8c71193c71ab6260be50"} Oct 31 15:45:19.850611 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:19.850289 4399 scope.go:122] "RemoveContainer" containerID="64d4a81c20898ca2be40df99beb6fac65b87184d29e34a335c61b5426f6db86b" Oct 31 15:45:19.850779 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:19.850729 4399 scope.go:122] "RemoveContainer" containerID="4495383f894e4049457cc85460f04e98f2304610fa4a8c71193c71ab6260be50" Oct 31 15:45:19.851057 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:45:19.851032 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq_odh-ai-gateway-infra(73e9cca1-362a-4150-af1f-4d36bb1e9e2d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" Oct 31 15:45:20.020934 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.020902 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:20.063324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.063291 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e062f91-e1dc-4644-830f-64054e2be83a-kube-api-access-f4ppk\" (UniqueName: \"kubernetes.io/projected/8e062f91-e1dc-4644-830f-64054e2be83a-kube-api-access-f4ppk\") pod \"8e062f91-e1dc-4644-830f-64054e2be83a\" (UID: \"8e062f91-e1dc-4644-830f-64054e2be83a\") " Oct 31 15:45:20.063492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.063439 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e062f91-e1dc-4644-830f-64054e2be83a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e062f91-e1dc-4644-830f-64054e2be83a-maas-api-service-ca\") pod \"8e062f91-e1dc-4644-830f-64054e2be83a\" (UID: \"8e062f91-e1dc-4644-830f-64054e2be83a\") " Oct 31 15:45:20.063794 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.063771 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e062f91-e1dc-4644-830f-64054e2be83a-maas-api-service-ca" pod "8e062f91-e1dc-4644-830f-64054e2be83a" (UID: "8e062f91-e1dc-4644-830f-64054e2be83a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:20.065792 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.065764 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e062f91-e1dc-4644-830f-64054e2be83a-kube-api-access-f4ppk" pod "8e062f91-e1dc-4644-830f-64054e2be83a" (UID: "8e062f91-e1dc-4644-830f-64054e2be83a"). InnerVolumeSpecName "kube-api-access-f4ppk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:20.077578 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.077547 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq"] Oct 31 15:45:20.187827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.187239 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e062f91-e1dc-4644-830f-64054e2be83a-maas-api-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:20.187827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.187281 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4ppk\" (UniqueName: \"kubernetes.io/projected/8e062f91-e1dc-4644-830f-64054e2be83a-kube-api-access-f4ppk\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:20.193947 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.190676 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wlhfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 31 15:45:20.193947 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.190768 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 31 15:45:20.858177 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.858142 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" Oct 31 15:45:20.858177 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.858162 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h" event={"ID":"8e062f91-e1dc-4644-830f-64054e2be83a","Type":"ContainerDied","Data":"e5246e9f356488c73af3d154a6a5462efe779c693d008606246ac6519bad8c36"} Oct 31 15:45:20.858684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:20.858270 4399 scope.go:122] "RemoveContainer" containerID="969c51d9fb8f88dc7c65a81f6faeb0cf6278894a03e1cfa1abaf0ceb2efe63a3" Oct 31 15:45:21.029921 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.029897 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:21.098475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.098434 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-maas-api-service-ca\") pod \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\" (UID: \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\") " Oct 31 15:45:21.098673 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.098523 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-kube-api-access-zlmkf\" (UniqueName: \"kubernetes.io/projected/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-kube-api-access-zlmkf\") pod \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\" (UID: \"73e9cca1-362a-4150-af1f-4d36bb1e9e2d\") " Oct 31 15:45:21.098935 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.098907 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-maas-api-service-ca" pod "73e9cca1-362a-4150-af1f-4d36bb1e9e2d" (UID: "73e9cca1-362a-4150-af1f-4d36bb1e9e2d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:21.099086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.099068 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-maas-api-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:21.100824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.100802 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-kube-api-access-zlmkf" pod "73e9cca1-362a-4150-af1f-4d36bb1e9e2d" (UID: "73e9cca1-362a-4150-af1f-4d36bb1e9e2d"). InnerVolumeSpecName "kube-api-access-zlmkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:21.138191 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.138160 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h"] Oct 31 15:45:21.141133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.141112 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-c947d9-29891025-sz72h"] Oct 31 15:45:21.200713 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.200678 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlmkf\" (UniqueName: \"kubernetes.io/projected/73e9cca1-362a-4150-af1f-4d36bb1e9e2d-kube-api-access-zlmkf\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:21.872612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.872572 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" Oct 31 15:45:21.872612 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.872603 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq" event={"ID":"73e9cca1-362a-4150-af1f-4d36bb1e9e2d","Type":"ContainerDied","Data":"04f559507bc2f9bf62c082be9f94b6398217486120532984aae46a90766dc698"} Oct 31 15:45:21.873227 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:21.872681 4399 scope.go:122] "RemoveContainer" containerID="4495383f894e4049457cc85460f04e98f2304610fa4a8c71193c71ab6260be50" Oct 31 15:45:22.036072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:22.036042 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq"] Oct 31 15:45:22.086116 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:22.086084 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-21111f-29891025-68pzq"] Oct 31 15:45:22.215486 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:22.215439 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-wlhfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:45:22.215665 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:22.215489 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:45:22.270586 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:22.270553 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:22.270997 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:22.270958 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:22.271801 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:22.271783 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:45:22.896749 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:22.896691 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" path="/var/lib/kubelet/pods/73e9cca1-362a-4150-af1f-4d36bb1e9e2d/volumes" Oct 31 15:45:22.897166 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:22.897136 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" path="/var/lib/kubelet/pods/8e062f91-e1dc-4644-830f-64054e2be83a/volumes" Oct 31 15:45:26.207228 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:45:26.207194 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:45:26.976887 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.976858 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2"] Oct 31 15:45:26.977513 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977494 4399 cpu_manager.go:414] "containerMap: removing container" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" containerName="cleanup" Oct 31 15:45:26.977623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977515 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" containerName="cleanup" Oct 31 15:45:26.977623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977527 4399 cpu_manager.go:414] "containerMap: removing container" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" containerName="cleanup" Oct 31 15:45:26.977623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977534 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" containerName="cleanup" Oct 31 15:45:26.977623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977555 4399 cpu_manager.go:414] "containerMap: removing container" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerName="cleanup" Oct 31 15:45:26.977623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977562 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerName="cleanup" Oct 31 15:45:26.977623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977589 4399 cpu_manager.go:414] "containerMap: removing container" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerName="cleanup" Oct 31 15:45:26.977623 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977596 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerName="cleanup" Oct 31 15:45:26.978050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977852 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" containerName="cleanup" Oct 31 15:45:26.978050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977867 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" containerName="cleanup" Oct 31 15:45:26.978050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977882 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerName="cleanup" Oct 31 15:45:26.978050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977892 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" containerName="cleanup" Oct 31 15:45:26.978050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977903 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerName="cleanup" Oct 31 15:45:26.978050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.977913 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerName="cleanup" Oct 31 15:45:26.982363 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:26.982343 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.026356 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.026323 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk"] Oct 31 15:45:27.026932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.026916 4399 cpu_manager.go:414] "containerMap: removing container" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" containerName="cleanup" Oct 31 15:45:27.027032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.026934 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e062f91-e1dc-4644-830f-64054e2be83a" containerName="cleanup" Oct 31 15:45:27.027032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.026956 4399 cpu_manager.go:414] "containerMap: removing container" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerName="cleanup" Oct 31 15:45:27.027032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.026963 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e9cca1-362a-4150-af1f-4d36bb1e9e2d" containerName="cleanup" Oct 31 15:45:27.030512 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.030494 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.081831 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.081802 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2"] Oct 31 15:45:27.088066 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.088020 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.088228 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.088096 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.088228 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.088170 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.088475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.088453 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.088548 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.088522 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.088609 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.088553 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.088677 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.088642 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.088780 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.088679 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.088780 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.088714 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v92bw\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-kube-api-access-v92bw\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.145559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.145527 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk"] Oct 31 15:45:27.190238 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190201 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.190415 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190249 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.190415 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190275 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v92bw\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-kube-api-access-v92bw\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.190415 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190349 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.190415 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190390 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.190415 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190411 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.190648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190441 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.190648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190578 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.190648 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190637 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.190881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190670 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.190881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190807 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.190881 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190846 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.191007 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190904 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.191007 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190926 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.191007 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190967 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.191007 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.190982 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.191133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.191039 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd2cl\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-kube-api-access-zd2cl\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.191304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.191285 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.191362 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.191345 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.191406 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.191391 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.191463 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.191424 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.191651 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.191630 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.192016 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.191949 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.194155 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.194119 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.194261 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.194244 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.246860 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.246788 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.247288 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.246962 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v92bw\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-kube-api-access-v92bw\") pod \"maas-default-gateway-openshift-default-67bf986f99-mt2j2\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.292233 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292194 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292233 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292240 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd2cl\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-kube-api-access-zd2cl\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292291 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292363 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292388 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292424 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292411 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292489 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292526 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292575 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292585 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292914 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292676 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.292914 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292783 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.293029 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.292915 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.293173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.293154 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.294965 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.294944 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.295158 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.295136 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.297043 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.297022 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:27.348499 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.348467 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.348650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.348517 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd2cl\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-kube-api-access-zd2cl\") pod \"maas-default-gateway-openshift-default-67bf986f99-cbzzk\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.644448 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.644360 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:27.730017 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:45:27.729986 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cad5cd_ecaf_4e90_9f88_9938f5c5d98d.slice/crio-3448ff4251fb41d0ca7569a40f88166078690a47c557ae2cec2743310995aa3c WatchSource:0}: Error finding container 3448ff4251fb41d0ca7569a40f88166078690a47c557ae2cec2743310995aa3c: Status 404 returned error can't find the container with id 3448ff4251fb41d0ca7569a40f88166078690a47c557ae2cec2743310995aa3c Oct 31 15:45:27.730017 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.729995 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2"] Oct 31 15:45:27.733140 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.733102 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:27.735847 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.733190 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:27.735847 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.733236 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:27.828027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.828003 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:45:27.828148 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.828064 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:45:27.925073 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.924991 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" event={"ID":"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d","Type":"ContainerStarted","Data":"aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe"} Oct 31 15:45:27.925073 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:27.925050 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" event={"ID":"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d","Type":"ContainerStarted","Data":"3448ff4251fb41d0ca7569a40f88166078690a47c557ae2cec2743310995aa3c"} Oct 31 15:45:28.025795 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.025769 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk"] Oct 31 15:45:28.027242 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:45:28.027213 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f1f8ea_ad57_49d8_b28b_b7a0d1e4dd15.slice/crio-d458befabb9f74e82dfa40332ec1983fc85438ebc741792021a76b8647e61c7e WatchSource:0}: Error finding container d458befabb9f74e82dfa40332ec1983fc85438ebc741792021a76b8647e61c7e: Status 404 returned error can't find the container with id d458befabb9f74e82dfa40332ec1983fc85438ebc741792021a76b8647e61c7e Oct 31 15:45:28.029998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.029967 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:28.030078 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.030028 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:28.030078 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.030057 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:28.188515 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.188400 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podStartSLOduration=2.188379842 podStartE2EDuration="2.188379842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:28.184118475 +0000 UTC m=+3571.395359890" watchObservedRunningTime="2026-10-31 15:45:28.188379842 +0000 UTC m=+3571.399621255" Oct 31 15:45:28.297291 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.297259 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:28.299092 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.299066 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mt2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 15:45:28.299211 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.299115 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 15:45:28.932492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.932457 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" event={"ID":"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15","Type":"ContainerStarted","Data":"c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd"} Oct 31 15:45:28.932669 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:28.932510 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" event={"ID":"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15","Type":"ContainerStarted","Data":"d458befabb9f74e82dfa40332ec1983fc85438ebc741792021a76b8647e61c7e"} Oct 31 15:45:29.298451 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:29.298401 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mt2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 15:45:29.298896 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:29.298477 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 15:45:29.660780 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:29.657679 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:29.664723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:29.662813 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cbzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:45:29.664723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:29.662884 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 15:45:30.298188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:30.297850 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mt2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 15:45:30.298188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:30.297929 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 15:45:30.648791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:30.645148 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cbzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:45:30.648791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:30.645220 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 15:45:31.299145 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:31.299108 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mt2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 15:45:31.299331 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:31.299156 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 15:45:31.649148 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:31.649060 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cbzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:45:31.663722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:31.650243 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 15:45:32.298092 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:32.298055 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mt2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 15:45:32.298279 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:32.298103 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 15:45:32.645778 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:32.645054 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cbzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:45:32.645778 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:32.645116 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 15:45:33.298750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:33.298475 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mt2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 15:45:33.298750 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:33.298549 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 15:45:33.646777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:33.645053 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cbzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:45:33.646777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:33.645113 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 15:45:34.298161 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:34.298117 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mt2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 15:45:34.298374 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:34.298182 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 15:45:34.644928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:34.644829 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cbzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:45:34.644928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:34.644895 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 15:45:35.298346 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:35.298304 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mt2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Oct 31 15:45:35.298513 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:35.298373 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Oct 31 15:45:35.645362 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:35.645273 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cbzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 31 15:45:35.645362 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:35.645339 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 31 15:45:37.338897 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.338853 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-mt2j2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:45:37.339431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.338920 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:45:37.453676 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.453620 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:37.454172 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.454142 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:37.455175 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.455155 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:45:37.674365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.674282 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-cbzzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:45:37.674365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.674351 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:45:37.694823 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.694769 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podStartSLOduration=11.69475509 podStartE2EDuration="11.69475509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:29.063009885 +0000 UTC m=+3572.274251338" watchObservedRunningTime="2026-10-31 15:45:37.69475509 +0000 UTC m=+3580.905996502" Oct 31 15:45:37.710650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.710621 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:37.711317 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.711236 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:37.711962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:37.711945 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:45:43.076816 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.076784 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx"] Oct 31 15:45:43.077309 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.077020 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" podUID="1d842564-cef1-48a8-b3c0-8b266a87375f" containerName="maas-api" containerID="cri-o://0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e" gracePeriod=30 Oct 31 15:45:43.402100 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.402076 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:45:43.476726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.473344 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d842564-cef1-48a8-b3c0-8b266a87375f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d842564-cef1-48a8-b3c0-8b266a87375f-openshift-service-ca\") pod \"1d842564-cef1-48a8-b3c0-8b266a87375f\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " Oct 31 15:45:43.476726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.473399 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-metrics-certs\") pod \"1d842564-cef1-48a8-b3c0-8b266a87375f\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " Oct 31 15:45:43.476726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.473541 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d842564-cef1-48a8-b3c0-8b266a87375f-kube-api-access-rcgrl\" (UniqueName: \"kubernetes.io/projected/1d842564-cef1-48a8-b3c0-8b266a87375f-kube-api-access-rcgrl\") pod \"1d842564-cef1-48a8-b3c0-8b266a87375f\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " Oct 31 15:45:43.476726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.473591 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls\") pod \"1d842564-cef1-48a8-b3c0-8b266a87375f\" (UID: \"1d842564-cef1-48a8-b3c0-8b266a87375f\") " Oct 31 15:45:43.478471 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.478431 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d842564-cef1-48a8-b3c0-8b266a87375f-openshift-service-ca" pod "1d842564-cef1-48a8-b3c0-8b266a87375f" (UID: "1d842564-cef1-48a8-b3c0-8b266a87375f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:45:43.486074 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.486038 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-metrics-certs" pod "1d842564-cef1-48a8-b3c0-8b266a87375f" (UID: "1d842564-cef1-48a8-b3c0-8b266a87375f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:45:43.498001 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.497171 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls" pod "1d842564-cef1-48a8-b3c0-8b266a87375f" (UID: "1d842564-cef1-48a8-b3c0-8b266a87375f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:45:43.507819 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.507673 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d842564-cef1-48a8-b3c0-8b266a87375f-kube-api-access-rcgrl" pod "1d842564-cef1-48a8-b3c0-8b266a87375f" (UID: "1d842564-cef1-48a8-b3c0-8b266a87375f"). InnerVolumeSpecName "kube-api-access-rcgrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:45:43.575745 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.575683 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d842564-cef1-48a8-b3c0-8b266a87375f-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:43.575745 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.575740 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:43.575909 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.575751 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcgrl\" (UniqueName: \"kubernetes.io/projected/1d842564-cef1-48a8-b3c0-8b266a87375f-kube-api-access-rcgrl\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:43.575909 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:43.575762 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1d842564-cef1-48a8-b3c0-8b266a87375f-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:45:44.090436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.090404 4399 generic.go:356] "Generic (PLEG): container finished" podID="1d842564-cef1-48a8-b3c0-8b266a87375f" containerID="0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e" exitCode=0 Oct 31 15:45:44.090903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.090442 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" Oct 31 15:45:44.090903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.090457 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" event={"ID":"1d842564-cef1-48a8-b3c0-8b266a87375f","Type":"ContainerDied","Data":"0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e"} Oct 31 15:45:44.090903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.090504 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx" event={"ID":"1d842564-cef1-48a8-b3c0-8b266a87375f","Type":"ContainerDied","Data":"2a56f178e0fdeee7a9e38d62117a56a56ead0f050fbe998932f41877b57e9834"} Oct 31 15:45:44.090903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.090529 4399 scope.go:122] "RemoveContainer" containerID="0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e" Oct 31 15:45:44.101082 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.101064 4399 scope.go:122] "RemoveContainer" containerID="0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e" Oct 31 15:45:44.101320 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:45:44.101303 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e\": container with ID starting with 0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e not found: ID does not exist" containerID="0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e" Oct 31 15:45:44.101366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.101328 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e"} err="failed to get container status \"0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e\": rpc error: code = NotFound desc = could not find container \"0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e\": container with ID starting with 0c5ae69aa5317c47974574576d70697d081dce6f3b9a071f1b59ae1593d96d8e not found: ID does not exist" Oct 31 15:45:44.253222 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.253185 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx"] Oct 31 15:45:44.302281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.302250 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-5a0cef-54bd564f7f-xlccx"] Oct 31 15:45:44.896685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:44.896653 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d842564-cef1-48a8-b3c0-8b266a87375f" path="/var/lib/kubelet/pods/1d842564-cef1-48a8-b3c0-8b266a87375f/volumes" Oct 31 15:45:51.852269 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.852236 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49"] Oct 31 15:45:51.853025 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.853007 4399 cpu_manager.go:414] "containerMap: removing container" podUID="1d842564-cef1-48a8-b3c0-8b266a87375f" containerName="maas-api" Oct 31 15:45:51.853094 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.853027 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d842564-cef1-48a8-b3c0-8b266a87375f" containerName="maas-api" Oct 31 15:45:51.853154 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.853145 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d842564-cef1-48a8-b3c0-8b266a87375f" containerName="maas-api" Oct 31 15:45:51.858259 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.858241 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:51.961675 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.958535 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49"] Oct 31 15:45:51.962925 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.962902 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-883697-openshift-default-dockercfg-dwjm8\"" Oct 31 15:45:51.990558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.988595 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-istio-token\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:51.990558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.988741 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f4b118-d329-4758-99d6-87367efd0468-istio-podinfo\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:51.990558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.988841 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f4b118-d329-4758-99d6-87367efd0468-istiod-ca-cert\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:51.990558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.988962 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-data\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:51.990558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.989060 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-certs\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:51.990558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.989497 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx46x\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-kube-api-access-kx46x\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:51.990558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.989690 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-envoy\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:51.990558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.989921 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-credential-socket\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:51.990558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:51.990332 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-socket\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.104769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.104348 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-envoy\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.104769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.104487 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-credential-socket\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.104769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.104512 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-socket\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.104769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.104601 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-istio-token\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.104769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.104639 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f4b118-d329-4758-99d6-87367efd0468-istio-podinfo\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.104769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.104732 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f4b118-d329-4758-99d6-87367efd0468-istiod-ca-cert\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.104769 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.104769 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-data\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.105199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.104836 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-certs\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.105199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.104917 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx46x\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-kube-api-access-kx46x\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.105199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.105017 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-socket\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.105199 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.105159 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-credential-socket\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.106061 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.105462 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-data\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.106061 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.105689 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f4b118-d329-4758-99d6-87367efd0468-istiod-ca-cert\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.106061 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.105822 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-certs\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.121748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.111015 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-envoy\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.121748 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.114036 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f4b118-d329-4758-99d6-87367efd0468-istio-podinfo\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.163734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.163662 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-istio-token\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.164037 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.164015 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx46x\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-kube-api-access-kx46x\") pod \"e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.172270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.172248 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:52.641530 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.641501 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49"] Oct 31 15:45:52.643246 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:45:52.643223 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f4b118_d329_4758_99d6_87367efd0468.slice/crio-f538d0c601a78c091b2eb5fcda259ec8681a87c41b06610c4e8205b0dacf7dea WatchSource:0}: Error finding container f538d0c601a78c091b2eb5fcda259ec8681a87c41b06610c4e8205b0dacf7dea: Status 404 returned error can't find the container with id f538d0c601a78c091b2eb5fcda259ec8681a87c41b06610c4e8205b0dacf7dea Oct 31 15:45:52.645825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.645794 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:52.645893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.645857 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:52.645893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:52.645886 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:45:53.151198 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:53.151165 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" event={"ID":"05f4b118-d329-4758-99d6-87367efd0468","Type":"ContainerStarted","Data":"accf8286ee975bb7cab62e4456f7e79c7d7da2cf5eec6027af536eac1e5dd349"} Oct 31 15:45:53.151660 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:53.151221 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" event={"ID":"05f4b118-d329-4758-99d6-87367efd0468","Type":"ContainerStarted","Data":"f538d0c601a78c091b2eb5fcda259ec8681a87c41b06610c4e8205b0dacf7dea"} Oct 31 15:45:53.173175 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:53.173150 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:53.179201 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:53.179180 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:53.266017 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:53.265976 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" podStartSLOduration=2.265961237 podStartE2EDuration="2.265961237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:45:53.264749783 +0000 UTC m=+3596.475991196" watchObservedRunningTime="2026-10-31 15:45:53.265961237 +0000 UTC m=+3596.477202648" Oct 31 15:45:54.157559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:54.157532 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:54.158541 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:45:54.158523 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:45:56.207227 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:45:56.207195 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:46:11.522318 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:11.522285 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6"] Oct 31 15:46:16.523270 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:16.523233 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" podUID="4d437b44-2517-4e0f-9870-a2c52a5c5839" containerName="payload-processing" containerID="cri-o://866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9" gracePeriod=25 Oct 31 15:46:16.802918 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:16.802893 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:46:16.820919 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:16.820897 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d437b44-2517-4e0f-9870-a2c52a5c5839-kube-api-access-xx9zc\" (UniqueName: \"kubernetes.io/projected/4d437b44-2517-4e0f-9870-a2c52a5c5839-kube-api-access-xx9zc\") pod \"4d437b44-2517-4e0f-9870-a2c52a5c5839\" (UID: \"4d437b44-2517-4e0f-9870-a2c52a5c5839\") " Oct 31 15:46:16.821045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:16.820993 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d437b44-2517-4e0f-9870-a2c52a5c5839-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d437b44-2517-4e0f-9870-a2c52a5c5839-plugins-config-volume\") pod \"4d437b44-2517-4e0f-9870-a2c52a5c5839\" (UID: \"4d437b44-2517-4e0f-9870-a2c52a5c5839\") " Oct 31 15:46:16.821343 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:16.821316 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d437b44-2517-4e0f-9870-a2c52a5c5839-plugins-config-volume" pod "4d437b44-2517-4e0f-9870-a2c52a5c5839" (UID: "4d437b44-2517-4e0f-9870-a2c52a5c5839"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:46:16.821596 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:16.821579 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d437b44-2517-4e0f-9870-a2c52a5c5839-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:16.823183 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:16.823161 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d437b44-2517-4e0f-9870-a2c52a5c5839-kube-api-access-xx9zc" pod "4d437b44-2517-4e0f-9870-a2c52a5c5839" (UID: "4d437b44-2517-4e0f-9870-a2c52a5c5839"). InnerVolumeSpecName "kube-api-access-xx9zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:46:16.923310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:16.923279 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx9zc\" (UniqueName: \"kubernetes.io/projected/4d437b44-2517-4e0f-9870-a2c52a5c5839-kube-api-access-xx9zc\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:46:17.321815 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:17.321783 4399 generic.go:356] "Generic (PLEG): container finished" podID="4d437b44-2517-4e0f-9870-a2c52a5c5839" containerID="866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9" exitCode=0 Oct 31 15:46:17.322004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:17.321832 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" Oct 31 15:46:17.322004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:17.321891 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" event={"ID":"4d437b44-2517-4e0f-9870-a2c52a5c5839","Type":"ContainerDied","Data":"866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9"} Oct 31 15:46:17.322004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:17.321962 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6" event={"ID":"4d437b44-2517-4e0f-9870-a2c52a5c5839","Type":"ContainerDied","Data":"fe5867bd6be28f0897cf6fec16f553fbc67fc2e4a7c26d3687250fe4253b38bf"} Oct 31 15:46:17.322004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:17.321998 4399 scope.go:122] "RemoveContainer" containerID="866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9" Oct 31 15:46:17.332456 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:17.332436 4399 scope.go:122] "RemoveContainer" containerID="866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9" Oct 31 15:46:17.332744 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:46:17.332722 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9\": container with ID starting with 866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9 not found: ID does not exist" containerID="866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9" Oct 31 15:46:17.332834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:17.332750 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9"} err="failed to get container status \"866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9\": rpc error: code = NotFound desc = could not find container \"866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9\": container with ID starting with 866e63a11b366baf777e57173c17f334ca70031e1ef190ad6d2f8cf9e759aca9 not found: ID does not exist" Oct 31 15:46:17.489428 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:17.489385 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6"] Oct 31 15:46:17.534899 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:17.534865 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-69517a-94665fb79-z5pf6"] Oct 31 15:46:18.896467 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:18.896437 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d437b44-2517-4e0f-9870-a2c52a5c5839" path="/var/lib/kubelet/pods/4d437b44-2517-4e0f-9870-a2c52a5c5839/volumes" Oct 31 15:46:26.206914 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:46:26.206879 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:46:31.253977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.253949 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt"] Oct 31 15:46:31.254536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.254523 4399 cpu_manager.go:414] "containerMap: removing container" podUID="4d437b44-2517-4e0f-9870-a2c52a5c5839" containerName="payload-processing" Oct 31 15:46:31.254582 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.254537 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d437b44-2517-4e0f-9870-a2c52a5c5839" containerName="payload-processing" Oct 31 15:46:31.254663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.254654 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d437b44-2517-4e0f-9870-a2c52a5c5839" containerName="payload-processing" Oct 31 15:46:31.258065 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.258047 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.317771 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.317734 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt"] Oct 31 15:46:31.318197 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.318174 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b18df8-serving-cert\"" Oct 31 15:46:31.394731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.394677 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.394893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.394862 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ktfx\" (UniqueName: \"kubernetes.io/projected/fd35a41e-27bd-4139-99fa-60f81f538369-kube-api-access-4ktfx\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.394936 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.394906 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-metrics-certs\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.394936 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.394922 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd35a41e-27bd-4139-99fa-60f81f538369-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.495777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.495739 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-metrics-certs\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.517891 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.517785 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd35a41e-27bd-4139-99fa-60f81f538369-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.518559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.518529 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.518973 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.518953 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ktfx\" (UniqueName: \"kubernetes.io/projected/fd35a41e-27bd-4139-99fa-60f81f538369-kube-api-access-4ktfx\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.534827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.520573 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd35a41e-27bd-4139-99fa-60f81f538369-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.535099 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:46:31.523249 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-serving-cert: secret "maas-api-e2e-inf-a-b18df8-serving-cert" not found Oct 31 15:46:31.535617 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:46:31.535597 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls podName:fd35a41e-27bd-4139-99fa-60f81f538369 nodeName:}" failed. No retries permitted until 2026-10-31 15:46:32.03557129 +0000 UTC m=+3635.246812688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls") pod "maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" (UID: "fd35a41e-27bd-4139-99fa-60f81f538369") : secret "maas-api-e2e-inf-a-b18df8-serving-cert" not found Oct 31 15:46:31.563433 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.563399 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-metrics-certs\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:31.586472 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:31.586393 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ktfx\" (UniqueName: \"kubernetes.io/projected/fd35a41e-27bd-4139-99fa-60f81f538369-kube-api-access-4ktfx\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:32.133093 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:32.133052 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:32.135399 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:32.135381 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls\") pod \"maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:32.172020 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:32.171994 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:32.549742 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:32.549679 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt"] Oct 31 15:46:32.551284 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:46:32.551253 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd35a41e_27bd_4139_99fa_60f81f538369.slice/crio-88301e6ccbb78889c22e09aa7181d7c2b291410050b883d879686043a6a04fa5 WatchSource:0}: Error finding container 88301e6ccbb78889c22e09aa7181d7c2b291410050b883d879686043a6a04fa5: Status 404 returned error can't find the container with id 88301e6ccbb78889c22e09aa7181d7c2b291410050b883d879686043a6a04fa5 Oct 31 15:46:32.552818 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:32.552800 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 15:46:33.443762 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:33.443725 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" event={"ID":"fd35a41e-27bd-4139-99fa-60f81f538369","Type":"ContainerStarted","Data":"88301e6ccbb78889c22e09aa7181d7c2b291410050b883d879686043a6a04fa5"} Oct 31 15:46:34.451848 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:34.451797 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" event={"ID":"fd35a41e-27bd-4139-99fa-60f81f538369","Type":"ContainerStarted","Data":"d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8"} Oct 31 15:46:34.452252 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:34.451987 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:34.569627 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:34.569569 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" podStartSLOduration=2.671861661 podStartE2EDuration="3.569554708s" totalImagesPullingTime="897.693047ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:46:34.566740443 +0000 UTC m=+3637.777981858" watchObservedRunningTime="2026-10-31 15:46:34.569554708 +0000 UTC m=+3637.780796119" Oct 31 15:46:40.465300 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:46:40.465107 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:46:56.207153 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:46:56.207110 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:47:26.207628 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:47:26.207592 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:47:38.916562 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:38.916478 4399 generic.go:356] "Generic (PLEG): container finished" podID="05f4b118-d329-4758-99d6-87367efd0468" containerID="accf8286ee975bb7cab62e4456f7e79c7d7da2cf5eec6027af536eac1e5dd349" exitCode=0 Oct 31 15:47:38.916562 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:38.916517 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" event={"ID":"05f4b118-d329-4758-99d6-87367efd0468","Type":"ContainerDied","Data":"accf8286ee975bb7cab62e4456f7e79c7d7da2cf5eec6027af536eac1e5dd349"} Oct 31 15:47:38.917152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:38.917055 4399 scope.go:122] "RemoveContainer" containerID="accf8286ee975bb7cab62e4456f7e79c7d7da2cf5eec6027af536eac1e5dd349" Oct 31 15:47:38.918760 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:38.918730 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:47:38.918873 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:38.918801 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:47:38.918873 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:38.918835 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:47:39.158141 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:39.158065 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:47:39.926175 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:39.926129 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" event={"ID":"05f4b118-d329-4758-99d6-87367efd0468","Type":"ContainerStarted","Data":"c6ebc117271d9caf50d4fe282a325148c26a7da45fee741a2863c276664220b3"} Oct 31 15:47:40.172874 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:40.172836 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:47:40.174559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:40.174523 4399 patch_prober.go:29] interesting pod/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:47:40.174771 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:40.174593 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 15:47:41.173600 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:41.173563 4399 patch_prober.go:29] interesting pod/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:47:41.174085 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:41.173622 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 15:47:42.172732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:42.172664 4399 patch_prober.go:29] interesting pod/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:47:42.172912 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:42.172785 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 15:47:43.172824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:43.172791 4399 patch_prober.go:29] interesting pod/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:47:43.173233 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:43.172843 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 15:47:44.173281 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:44.173239 4399 patch_prober.go:29] interesting pod/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 31 15:47:44.173727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:44.173304 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 31 15:47:45.177369 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:45.177340 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:47:45.177805 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:45.177777 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:47:45.178431 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:47:45.178412 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:47:56.208831 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:47:56.208793 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:48:09.760902 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:09.760687 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz"] Oct 31 15:48:09.786749 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:09.786210 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:09.831764 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:09.831730 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz"] Oct 31 15:48:09.876690 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:09.876662 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c4840c\"" Oct 31 15:48:09.876902 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:09.876662 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c4840c-dockercfg-hrtk7\"" Oct 31 15:48:09.976230 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:09.976199 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmdrz\" (UniqueName: \"kubernetes.io/projected/363d1acf-1182-4f18-855a-8b8b8fca41ed-kube-api-access-rmdrz\") pod \"payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz\" (UID: \"363d1acf-1182-4f18-855a-8b8b8fca41ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:09.976410 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:09.976329 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363d1acf-1182-4f18-855a-8b8b8fca41ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz\" (UID: \"363d1acf-1182-4f18-855a-8b8b8fca41ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:10.090656 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:10.090544 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363d1acf-1182-4f18-855a-8b8b8fca41ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz\" (UID: \"363d1acf-1182-4f18-855a-8b8b8fca41ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:10.106329 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:10.106293 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmdrz\" (UniqueName: \"kubernetes.io/projected/363d1acf-1182-4f18-855a-8b8b8fca41ed-kube-api-access-rmdrz\") pod \"payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz\" (UID: \"363d1acf-1182-4f18-855a-8b8b8fca41ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:10.107072 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:10.093073 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363d1acf-1182-4f18-855a-8b8b8fca41ed-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz\" (UID: \"363d1acf-1182-4f18-855a-8b8b8fca41ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:10.163743 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:10.163685 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmdrz\" (UniqueName: \"kubernetes.io/projected/363d1acf-1182-4f18-855a-8b8b8fca41ed-kube-api-access-rmdrz\") pod \"payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz\" (UID: \"363d1acf-1182-4f18-855a-8b8b8fca41ed\") " pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:10.408144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:10.408069 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:10.783566 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:10.783538 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz"] Oct 31 15:48:10.785630 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:48:10.785593 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363d1acf_1182_4f18_855a_8b8b8fca41ed.slice/crio-224fb002dc6eadf532e65b171121f58c2e7ae2a7db853b12b471b783d7022476 WatchSource:0}: Error finding container 224fb002dc6eadf532e65b171121f58c2e7ae2a7db853b12b471b783d7022476: Status 404 returned error can't find the container with id 224fb002dc6eadf532e65b171121f58c2e7ae2a7db853b12b471b783d7022476 Oct 31 15:48:11.186553 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:11.186440 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" event={"ID":"363d1acf-1182-4f18-855a-8b8b8fca41ed","Type":"ContainerStarted","Data":"2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4"} Oct 31 15:48:11.186553 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:11.186527 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" event={"ID":"363d1acf-1182-4f18-855a-8b8b8fca41ed","Type":"ContainerStarted","Data":"224fb002dc6eadf532e65b171121f58c2e7ae2a7db853b12b471b783d7022476"} Oct 31 15:48:11.186553 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:11.186550 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:11.304030 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:11.303977 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" podStartSLOduration=2.3039630190000002 podStartE2EDuration="2.303963019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:48:11.298022312 +0000 UTC m=+3734.509263724" watchObservedRunningTime="2026-10-31 15:48:11.303963019 +0000 UTC m=+3734.515204430" Oct 31 15:48:22.196825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:22.196794 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:48:26.207127 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:48:26.207089 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:48:53.554213 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.554181 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5"] Oct 31 15:48:53.559486 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.559459 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:48:53.649723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.647507 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5\" (UID: \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:48:53.649723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.647622 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhs8\" (UniqueName: \"kubernetes.io/projected/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-kube-api-access-zlhs8\") pod \"payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5\" (UID: \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:48:53.664068 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.664037 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cf95c9-dockercfg-wj2l5\"" Oct 31 15:48:53.664397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.664377 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cf95c9\"" Oct 31 15:48:53.752205 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.752168 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5"] Oct 31 15:48:53.753467 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.753434 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5\" (UID: \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:48:53.753690 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.753675 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhs8\" (UniqueName: \"kubernetes.io/projected/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-kube-api-access-zlhs8\") pod \"payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5\" (UID: \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:48:53.755092 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.755070 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5\" (UID: \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:48:53.812631 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.812567 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhs8\" (UniqueName: \"kubernetes.io/projected/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-kube-api-access-zlhs8\") pod \"payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5\" (UID: \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:48:53.875970 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:53.875946 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:48:54.251634 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:54.251609 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5"] Oct 31 15:48:54.253509 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:48:54.253473 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0a4375_4680_4d0e_a70d_37151b2eb5f6.slice/crio-b05f93cf0f27a5f86f7884d5d5d83a0104bb8ddcfe12eb22f3758eb6c7628cf4 WatchSource:0}: Error finding container b05f93cf0f27a5f86f7884d5d5d83a0104bb8ddcfe12eb22f3758eb6c7628cf4: Status 404 returned error can't find the container with id b05f93cf0f27a5f86f7884d5d5d83a0104bb8ddcfe12eb22f3758eb6c7628cf4 Oct 31 15:48:54.519494 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:54.519408 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" event={"ID":"fa0a4375-4680-4d0e-a70d-37151b2eb5f6","Type":"ContainerStarted","Data":"28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612"} Oct 31 15:48:54.519494 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:54.519469 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" event={"ID":"fa0a4375-4680-4d0e-a70d-37151b2eb5f6","Type":"ContainerStarted","Data":"b05f93cf0f27a5f86f7884d5d5d83a0104bb8ddcfe12eb22f3758eb6c7628cf4"} Oct 31 15:48:54.519757 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:54.519510 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:48:54.635868 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:48:54.635816 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" podStartSLOduration=1.635802507 podStartE2EDuration="1.635802507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:48:54.630809298 +0000 UTC m=+3777.842050712" watchObservedRunningTime="2026-10-31 15:48:54.635802507 +0000 UTC m=+3777.847043921" Oct 31 15:48:56.207242 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:48:56.207210 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:49:05.529134 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:05.529107 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:49:07.283481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.283450 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445"] Oct 31 15:49:07.287957 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.287933 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.345446 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.345409 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445"] Oct 31 15:49:07.389251 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.389211 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b21d8a-serving-cert\"" Oct 31 15:49:07.442628 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.442593 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-metrics-certs\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.442823 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.442635 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bw2m\" (UniqueName: \"kubernetes.io/projected/29f054d2-d69a-46eb-957c-d1d46b39bbfe-kube-api-access-4bw2m\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.442823 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.442755 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-maas-api-tls\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.442964 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.442829 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f054d2-d69a-46eb-957c-d1d46b39bbfe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.558515 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.556112 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-maas-api-tls\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.558515 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.556286 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f054d2-d69a-46eb-957c-d1d46b39bbfe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.590731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.578918 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-metrics-certs\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.590731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.578973 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bw2m\" (UniqueName: \"kubernetes.io/projected/29f054d2-d69a-46eb-957c-d1d46b39bbfe-kube-api-access-4bw2m\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.590731 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.563187 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f054d2-d69a-46eb-957c-d1d46b39bbfe-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.591734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.591655 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-maas-api-tls\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.604734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.603898 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-metrics-certs\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.635395 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.635363 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bw2m\" (UniqueName: \"kubernetes.io/projected/29f054d2-d69a-46eb-957c-d1d46b39bbfe-kube-api-access-4bw2m\") pod \"maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:07.901905 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:07.901833 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:08.277555 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:08.277534 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445"] Oct 31 15:49:08.279801 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:49:08.279771 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f054d2_d69a_46eb_957c_d1d46b39bbfe.slice/crio-c602558a00c44a2761ba4d71fa6e53a20c85143a1662ff37d0aa9d5dc742c90c WatchSource:0}: Error finding container c602558a00c44a2761ba4d71fa6e53a20c85143a1662ff37d0aa9d5dc742c90c: Status 404 returned error can't find the container with id c602558a00c44a2761ba4d71fa6e53a20c85143a1662ff37d0aa9d5dc742c90c Oct 31 15:49:08.616981 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:08.616900 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" event={"ID":"29f054d2-d69a-46eb-957c-d1d46b39bbfe","Type":"ContainerStarted","Data":"c602558a00c44a2761ba4d71fa6e53a20c85143a1662ff37d0aa9d5dc742c90c"} Oct 31 15:49:09.624858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:09.624823 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" event={"ID":"29f054d2-d69a-46eb-957c-d1d46b39bbfe","Type":"ContainerStarted","Data":"a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697"} Oct 31 15:49:09.625256 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:09.624961 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:09.741397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:09.741346 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" podStartSLOduration=1.877829781 podStartE2EDuration="2.741333108s" totalImagesPullingTime="863.503327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:49:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:49:09.738311692 +0000 UTC m=+3792.949553105" watchObservedRunningTime="2026-10-31 15:49:09.741333108 +0000 UTC m=+3792.952574519" Oct 31 15:49:15.636894 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:15.636868 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:49:26.206188 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:49:26.206146 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:49:32.325164 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.325129 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk"] Oct 31 15:49:32.330224 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.330204 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.386035 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.386002 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk"] Oct 31 15:49:32.426451 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.426403 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wptf\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-kube-api-access-9wptf\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.426614 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.426539 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-token\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.426614 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.426586 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-envoy\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.426817 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.426681 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-socket\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.426817 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.426748 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83e1c67-842d-4759-8e56-7d14d56f4e51-istiod-ca-cert\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.426933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.426821 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-podinfo\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.426933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.426868 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-credential-socket\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.427045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.426975 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-certs\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.427107 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.427046 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-data\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.431141 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.431117 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c5ee32-openshift-default-dockercfg-nghgs\"" Oct 31 15:49:32.528170 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528142 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-envoy\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.528345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528204 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-socket\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.528345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528231 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83e1c67-842d-4759-8e56-7d14d56f4e51-istiod-ca-cert\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.528345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528262 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-podinfo\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.528345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528303 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-credential-socket\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.528345 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528344 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-certs\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.528647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528396 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-data\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.528647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528443 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wptf\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-kube-api-access-9wptf\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.528647 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528550 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-token\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.528850 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.528651 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-socket\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.529055 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.529026 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-data\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.529178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.529059 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-credential-socket\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.529178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.529087 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83e1c67-842d-4759-8e56-7d14d56f4e51-istiod-ca-cert\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.529178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.529087 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-certs\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.531112 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.531086 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-podinfo\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.531210 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.531150 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-envoy\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.585047 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.584984 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-token\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.585144 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.585072 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wptf\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-kube-api-access-9wptf\") pod \"e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:32.643497 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:32.643472 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:33.109257 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:33.109231 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk"] Oct 31 15:49:33.110763 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:49:33.110739 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83e1c67_842d_4759_8e56_7d14d56f4e51.slice/crio-a540a7486d56b9a3893e87e8877a28ae68a54492eded3813030dbe4abc7002f2 WatchSource:0}: Error finding container a540a7486d56b9a3893e87e8877a28ae68a54492eded3813030dbe4abc7002f2: Status 404 returned error can't find the container with id a540a7486d56b9a3893e87e8877a28ae68a54492eded3813030dbe4abc7002f2 Oct 31 15:49:33.113433 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:33.113399 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:49:33.113504 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:33.113468 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:49:33.113542 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:33.113503 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:49:33.808547 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:33.808512 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" event={"ID":"f83e1c67-842d-4759-8e56-7d14d56f4e51","Type":"ContainerStarted","Data":"e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1"} Oct 31 15:49:33.809089 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:33.808612 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" event={"ID":"f83e1c67-842d-4759-8e56-7d14d56f4e51","Type":"ContainerStarted","Data":"a540a7486d56b9a3893e87e8877a28ae68a54492eded3813030dbe4abc7002f2"} Oct 31 15:49:33.926317 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:33.926258 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" podStartSLOduration=1.926244737 podStartE2EDuration="1.926244737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:49:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:49:33.923000843 +0000 UTC m=+3817.134242255" watchObservedRunningTime="2026-10-31 15:49:33.926244737 +0000 UTC m=+3817.137486150" Oct 31 15:49:34.644045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:34.644015 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:34.649977 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:34.649950 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:34.815534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:34.815502 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:34.816428 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:49:34.816403 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:49:56.234931 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:49:56.234889 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:50:15.045041 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:15.045004 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg"] Oct 31 15:50:15.104928 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:15.104892 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1e34efa-gvtqg"] Oct 31 15:50:16.899276 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:16.899245 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b506a697-275d-45ee-8922-521497e1880c" path="/var/lib/kubelet/pods/b506a697-275d-45ee-8922-521497e1880c/volumes" Oct 31 15:50:26.207368 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:26.207331 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:50:28.765249 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.765216 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg"] Oct 31 15:50:28.770824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.770802 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.825210 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.825172 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.825340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.825274 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwsvq\" (UniqueName: \"kubernetes.io/projected/12a04ce2-d74b-4cfb-87e9-36f916d4f590-kube-api-access-nwsvq\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.825340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.825305 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a04ce2-d74b-4cfb-87e9-36f916d4f590-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.825464 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.825413 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-metrics-certs\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.827353 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.827330 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg"] Oct 31 15:50:28.871969 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.871939 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cf95c9-serving-cert\"" Oct 31 15:50:28.926364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.926332 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-metrics-certs\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.926563 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.926453 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.926636 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:28.926563 4399 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-serving-cert: secret "maas-api-e2e-inf-b-cf95c9-serving-cert" not found Oct 31 15:50:28.926636 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.926591 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwsvq\" (UniqueName: \"kubernetes.io/projected/12a04ce2-d74b-4cfb-87e9-36f916d4f590-kube-api-access-nwsvq\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.926636 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:28.926626 4399 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls podName:12a04ce2-d74b-4cfb-87e9-36f916d4f590 nodeName:}" failed. No retries permitted until 2026-10-31 15:50:29.426604973 +0000 UTC m=+3872.637846380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls") pod "maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" (UID: "12a04ce2-d74b-4cfb-87e9-36f916d4f590") : secret "maas-api-e2e-inf-b-cf95c9-serving-cert" not found Oct 31 15:50:28.926818 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.926742 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a04ce2-d74b-4cfb-87e9-36f916d4f590-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.927324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.927305 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a04ce2-d74b-4cfb-87e9-36f916d4f590-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.928925 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.928904 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-metrics-certs\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:28.983216 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:28.983190 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwsvq\" (UniqueName: \"kubernetes.io/projected/12a04ce2-d74b-4cfb-87e9-36f916d4f590-kube-api-access-nwsvq\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:29.461054 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:29.461018 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:29.463586 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:29.463558 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls\") pod \"maas-api-e2e-inf-b-cf95c9-557db449-5ljlg\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:29.685109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:29.685071 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:30.060358 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:30.060330 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg"] Oct 31 15:50:30.062335 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:50:30.062310 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a04ce2_d74b_4cfb_87e9_36f916d4f590.slice/crio-654fded306ab79d2752e2fc1d1af506679ced29df92fa244126d77f1e9bea300 WatchSource:0}: Error finding container 654fded306ab79d2752e2fc1d1af506679ced29df92fa244126d77f1e9bea300: Status 404 returned error can't find the container with id 654fded306ab79d2752e2fc1d1af506679ced29df92fa244126d77f1e9bea300 Oct 31 15:50:30.189869 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:30.189826 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" event={"ID":"12a04ce2-d74b-4cfb-87e9-36f916d4f590","Type":"ContainerStarted","Data":"654fded306ab79d2752e2fc1d1af506679ced29df92fa244126d77f1e9bea300"} Oct 31 15:50:31.197101 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:31.197066 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" event={"ID":"12a04ce2-d74b-4cfb-87e9-36f916d4f590","Type":"ContainerStarted","Data":"08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058"} Oct 31 15:50:31.197535 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:31.197140 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:31.310496 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:31.310448 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" podStartSLOduration=2.461647052 podStartE2EDuration="3.310433641s" totalImagesPullingTime="848.786589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:50:31.308782047 +0000 UTC m=+3874.520023459" watchObservedRunningTime="2026-10-31 15:50:31.310433641 +0000 UTC m=+3874.521675074" Oct 31 15:50:37.209040 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:37.209004 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:50:44.309084 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:44.309049 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk"] Oct 31 15:50:44.309567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:44.309253 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" containerID="cri-o://c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd" gracePeriod=30 Oct 31 15:50:44.358685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:44.358649 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2"] Oct 31 15:50:44.358959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:44.358930 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" containerID="cri-o://aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe" gracePeriod=30 Oct 31 15:50:44.361838 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:44.361809 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl"] Oct 31 15:50:44.362021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:44.361997 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" containerID="cri-o://c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127" gracePeriod=30 Oct 31 15:50:44.408441 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:44.408401 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv"] Oct 31 15:50:44.408639 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:44.408617 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" containerID="cri-o://cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc" gracePeriod=30 Oct 31 15:50:47.331142 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.331104 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445"] Oct 31 15:50:47.331982 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.331950 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" podUID="29f054d2-d69a-46eb-957c-d1d46b39bbfe" containerName="maas-api" containerID="cri-o://a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697" gracePeriod=30 Oct 31 15:50:47.414037 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:47.413992 4399 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f054d2_d69a_46eb_957c_d1d46b39bbfe.slice/crio-conmon-a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f054d2_d69a_46eb_957c_d1d46b39bbfe.slice/crio-a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697.scope\": RecentStats: unable to find data in memory cache]" Oct 31 15:50:47.414224 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:47.414189 4399 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f054d2_d69a_46eb_957c_d1d46b39bbfe.slice/crio-a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29f054d2_d69a_46eb_957c_d1d46b39bbfe.slice/crio-conmon-a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697.scope\": RecentStats: unable to find data in memory cache]" Oct 31 15:50:47.656941 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.656913 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:50:47.805682 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.805649 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-maas-api-tls\") pod \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " Oct 31 15:50:47.805906 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.805788 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29f054d2-d69a-46eb-957c-d1d46b39bbfe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f054d2-d69a-46eb-957c-d1d46b39bbfe-openshift-service-ca\") pod \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " Oct 31 15:50:47.805990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.805910 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29f054d2-d69a-46eb-957c-d1d46b39bbfe-kube-api-access-4bw2m\" (UniqueName: \"kubernetes.io/projected/29f054d2-d69a-46eb-957c-d1d46b39bbfe-kube-api-access-4bw2m\") pod \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " Oct 31 15:50:47.805990 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.805942 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-metrics-certs\") pod \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\" (UID: \"29f054d2-d69a-46eb-957c-d1d46b39bbfe\") " Oct 31 15:50:47.806234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.806204 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29f054d2-d69a-46eb-957c-d1d46b39bbfe-openshift-service-ca" pod "29f054d2-d69a-46eb-957c-d1d46b39bbfe" (UID: "29f054d2-d69a-46eb-957c-d1d46b39bbfe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:50:47.806820 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.806794 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/29f054d2-d69a-46eb-957c-d1d46b39bbfe-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:47.808223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.808200 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29f054d2-d69a-46eb-957c-d1d46b39bbfe-kube-api-access-4bw2m" pod "29f054d2-d69a-46eb-957c-d1d46b39bbfe" (UID: "29f054d2-d69a-46eb-957c-d1d46b39bbfe"). InnerVolumeSpecName "kube-api-access-4bw2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:47.808439 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.808410 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-maas-api-tls" pod "29f054d2-d69a-46eb-957c-d1d46b39bbfe" (UID: "29f054d2-d69a-46eb-957c-d1d46b39bbfe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:50:47.808614 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.808596 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-metrics-certs" pod "29f054d2-d69a-46eb-957c-d1d46b39bbfe" (UID: "29f054d2-d69a-46eb-957c-d1d46b39bbfe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:50:47.908341 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.908263 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bw2m\" (UniqueName: \"kubernetes.io/projected/29f054d2-d69a-46eb-957c-d1d46b39bbfe-kube-api-access-4bw2m\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:47.908341 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.908294 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:47.908341 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:47.908308 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/29f054d2-d69a-46eb-957c-d1d46b39bbfe-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:48.329879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.326671 4399 generic.go:356] "Generic (PLEG): container finished" podID="29f054d2-d69a-46eb-957c-d1d46b39bbfe" containerID="a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697" exitCode=0 Oct 31 15:50:48.329879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.326917 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" event={"ID":"29f054d2-d69a-46eb-957c-d1d46b39bbfe","Type":"ContainerDied","Data":"a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697"} Oct 31 15:50:48.329879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.326974 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" event={"ID":"29f054d2-d69a-46eb-957c-d1d46b39bbfe","Type":"ContainerDied","Data":"c602558a00c44a2761ba4d71fa6e53a20c85143a1662ff37d0aa9d5dc742c90c"} Oct 31 15:50:48.329879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.327006 4399 scope.go:122] "RemoveContainer" containerID="a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697" Oct 31 15:50:48.329879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.327182 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445" Oct 31 15:50:48.352821 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.352660 4399 scope.go:122] "RemoveContainer" containerID="a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697" Oct 31 15:50:48.353074 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:48.353028 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697\": container with ID starting with a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697 not found: ID does not exist" containerID="a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697" Oct 31 15:50:48.353074 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.353054 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697"} err="failed to get container status \"a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697\": rpc error: code = NotFound desc = could not find container \"a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697\": container with ID starting with a3f361e677da511eada3242296e9bbb29c7027448b11d08bc45ef6f05fb11697 not found: ID does not exist" Oct 31 15:50:48.494223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.494194 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445"] Oct 31 15:50:48.498387 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.498365 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b21d8a-6cd5b655d7-54445"] Oct 31 15:50:48.897183 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:48.897154 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29f054d2-d69a-46eb-957c-d1d46b39bbfe" path="/var/lib/kubelet/pods/29f054d2-d69a-46eb-957c-d1d46b39bbfe/volumes" Oct 31 15:50:49.664738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.662893 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:50:49.725304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.725279 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:50:49.777913 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.775460 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:50:49.795852 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.795821 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:50:49.853903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.853802 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-envoy\") pod \"ba41a32a-6505-47b2-9fb7-842b885e8369\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " Oct 31 15:50:49.853903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.853856 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-credential-socket\") pod \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " Oct 31 15:50:49.853903 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.853900 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-podinfo\") pod \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " Oct 31 15:50:49.854136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.853957 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-socket\") pod \"ba41a32a-6505-47b2-9fb7-842b885e8369\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " Oct 31 15:50:49.854136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.853990 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-data\") pod \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " Oct 31 15:50:49.854136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854029 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istiod-ca-cert\") pod \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " Oct 31 15:50:49.854136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854076 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-token\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-token\") pod \"ba41a32a-6505-47b2-9fb7-842b885e8369\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " Oct 31 15:50:49.854136 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854104 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-certs\") pod \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " Oct 31 15:50:49.854298 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854192 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-kube-api-access-gfs4g\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-kube-api-access-gfs4g\") pod \"ba41a32a-6505-47b2-9fb7-842b885e8369\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " Oct 31 15:50:49.854298 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854217 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-socket\") pod \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " Oct 31 15:50:49.854298 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854265 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-kube-api-access-zd2cl\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-kube-api-access-zd2cl\") pod \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " Oct 31 15:50:49.854298 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854260 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-socket" pod "ba41a32a-6505-47b2-9fb7-842b885e8369" (UID: "ba41a32a-6505-47b2-9fb7-842b885e8369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.854491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854301 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba41a32a-6505-47b2-9fb7-842b885e8369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba41a32a-6505-47b2-9fb7-842b885e8369-istiod-ca-cert\") pod \"ba41a32a-6505-47b2-9fb7-842b885e8369\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " Oct 31 15:50:49.854491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854357 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-podinfo\") pod \"ba41a32a-6505-47b2-9fb7-842b885e8369\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " Oct 31 15:50:49.854491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854407 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-token\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-token\") pod \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " Oct 31 15:50:49.854491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854476 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-credential-socket\") pod \"ba41a32a-6505-47b2-9fb7-842b885e8369\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " Oct 31 15:50:49.854491 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854475 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-certs" pod "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" (UID: "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.854788 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854516 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-data\") pod \"ba41a32a-6505-47b2-9fb7-842b885e8369\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " Oct 31 15:50:49.854788 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854508 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-socket" pod "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" (UID: "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.854788 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854552 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-envoy\") pod \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\" (UID: \"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15\") " Oct 31 15:50:49.854788 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854642 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-certs\") pod \"ba41a32a-6505-47b2-9fb7-842b885e8369\" (UID: \"ba41a32a-6505-47b2-9fb7-842b885e8369\") " Oct 31 15:50:49.854788 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854644 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-data" pod "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" (UID: "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.854788 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854753 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istiod-ca-cert" pod "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" (UID: "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:50:49.855105 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.854952 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-credential-socket" pod "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" (UID: "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.855516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855209 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-certs" pod "ba41a32a-6505-47b2-9fb7-842b885e8369" (UID: "ba41a32a-6505-47b2-9fb7-842b885e8369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.855516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855346 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-data" pod "ba41a32a-6505-47b2-9fb7-842b885e8369" (UID: "ba41a32a-6505-47b2-9fb7-842b885e8369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.855516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855400 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.855516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855412 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-credential-socket" pod "ba41a32a-6505-47b2-9fb7-842b885e8369" (UID: "ba41a32a-6505-47b2-9fb7-842b885e8369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.855516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855422 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.855516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855479 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.855516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855496 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.855516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855508 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.855516 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855522 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.856311 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855536 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.856311 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.855654 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba41a32a-6505-47b2-9fb7-842b885e8369-istiod-ca-cert" pod "ba41a32a-6505-47b2-9fb7-842b885e8369" (UID: "ba41a32a-6505-47b2-9fb7-842b885e8369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:50:49.858313 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.858285 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-envoy" pod "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" (UID: "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.860524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.860498 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-podinfo" pod "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" (UID: "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:50:49.860984 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.860848 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-kube-api-access-zd2cl" pod "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" (UID: "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15"). InnerVolumeSpecName "kube-api-access-zd2cl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:49.861128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.861032 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-podinfo" pod "ba41a32a-6505-47b2-9fb7-842b885e8369" (UID: "ba41a32a-6505-47b2-9fb7-842b885e8369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:50:49.861344 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.861316 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-envoy" pod "ba41a32a-6505-47b2-9fb7-842b885e8369" (UID: "ba41a32a-6505-47b2-9fb7-842b885e8369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.861955 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.861926 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-token" pod "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" (UID: "e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:49.862217 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.862194 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-kube-api-access-gfs4g" pod "ba41a32a-6505-47b2-9fb7-842b885e8369" (UID: "ba41a32a-6505-47b2-9fb7-842b885e8369"). InnerVolumeSpecName "kube-api-access-gfs4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:49.863086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.863062 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-token" pod "ba41a32a-6505-47b2-9fb7-842b885e8369" (UID: "ba41a32a-6505-47b2-9fb7-842b885e8369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:49.956782 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.956748 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-data\") pod \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " Oct 31 15:50:49.956956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.956821 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istiod-ca-cert\") pod \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " Oct 31 15:50:49.956956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.956858 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-socket\") pod \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " Oct 31 15:50:49.956956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.956904 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-token\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-token\") pod \"9db214e1-a5bb-414f-b55a-23b0130f9115\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " Oct 31 15:50:49.956956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.956931 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-podinfo\") pod \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " Oct 31 15:50:49.957195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957006 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-credential-socket\") pod \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " Oct 31 15:50:49.957195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957053 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-envoy\") pod \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " Oct 31 15:50:49.957195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957085 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-socket\") pod \"9db214e1-a5bb-414f-b55a-23b0130f9115\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " Oct 31 15:50:49.957195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957115 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-envoy\") pod \"9db214e1-a5bb-414f-b55a-23b0130f9115\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " Oct 31 15:50:49.957195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957145 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-data\") pod \"9db214e1-a5bb-414f-b55a-23b0130f9115\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " Oct 31 15:50:49.957195 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957189 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-kube-api-access-v92bw\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-kube-api-access-v92bw\") pod \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " Oct 31 15:50:49.957511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957200 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-socket" pod "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" (UID: "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.957511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957241 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-certs\") pod \"9db214e1-a5bb-414f-b55a-23b0130f9115\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " Oct 31 15:50:49.957511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957268 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-certs\") pod \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " Oct 31 15:50:49.957511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957307 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istiod-ca-cert" pod "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" (UID: "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:50:49.957511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957349 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-podinfo\") pod \"9db214e1-a5bb-414f-b55a-23b0130f9115\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " Oct 31 15:50:49.957511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957359 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-credential-socket" pod "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" (UID: "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.957511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957391 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-credential-socket\") pod \"9db214e1-a5bb-414f-b55a-23b0130f9115\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " Oct 31 15:50:49.957511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957499 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-socket" pod "9db214e1-a5bb-414f-b55a-23b0130f9115" (UID: "9db214e1-a5bb-414f-b55a-23b0130f9115"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.957511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957508 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-token\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-token\") pod \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\" (UID: \"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d\") " Oct 31 15:50:49.958011 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957563 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db214e1-a5bb-414f-b55a-23b0130f9115-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db214e1-a5bb-414f-b55a-23b0130f9115-istiod-ca-cert\") pod \"9db214e1-a5bb-414f-b55a-23b0130f9115\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " Oct 31 15:50:49.958011 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957602 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-kube-api-access-87x8b\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-kube-api-access-87x8b\") pod \"9db214e1-a5bb-414f-b55a-23b0130f9115\" (UID: \"9db214e1-a5bb-414f-b55a-23b0130f9115\") " Oct 31 15:50:49.958011 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957870 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-certs" pod "9db214e1-a5bb-414f-b55a-23b0130f9115" (UID: "9db214e1-a5bb-414f-b55a-23b0130f9115"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.958011 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957992 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-certs" pod "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" (UID: "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958309 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db214e1-a5bb-414f-b55a-23b0130f9115-istiod-ca-cert" pod "9db214e1-a5bb-414f-b55a-23b0130f9115" (UID: "9db214e1-a5bb-414f-b55a-23b0130f9115"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958318 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.957083 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-data" pod "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" (UID: "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958345 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958361 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958374 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958389 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958403 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958417 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958431 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958444 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958456 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958470 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958483 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958497 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958510 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.958524 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958523 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfs4g\" (UniqueName: \"kubernetes.io/projected/ba41a32a-6505-47b2-9fb7-842b885e8369-kube-api-access-gfs4g\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.963027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958539 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd2cl\" (UniqueName: \"kubernetes.io/projected/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15-kube-api-access-zd2cl\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.963027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958535 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-data" pod "9db214e1-a5bb-414f-b55a-23b0130f9115" (UID: "9db214e1-a5bb-414f-b55a-23b0130f9115"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.963027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958551 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba41a32a-6505-47b2-9fb7-842b885e8369-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:49.963027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.958687 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-credential-socket" pod "9db214e1-a5bb-414f-b55a-23b0130f9115" (UID: "9db214e1-a5bb-414f-b55a-23b0130f9115"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.963027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.962075 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-podinfo" pod "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" (UID: "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:50:49.963027 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.962175 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-token" pod "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" (UID: "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:49.963774 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.963739 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-kube-api-access-87x8b" pod "9db214e1-a5bb-414f-b55a-23b0130f9115" (UID: "9db214e1-a5bb-414f-b55a-23b0130f9115"). InnerVolumeSpecName "kube-api-access-87x8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:49.964128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.964086 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-envoy" pod "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" (UID: "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.964128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.964102 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-kube-api-access-v92bw" pod "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" (UID: "35cad5cd-ecaf-4e90-9f88-9938f5c5d98d"). InnerVolumeSpecName "kube-api-access-v92bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:49.964620 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.964591 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-token" pod "9db214e1-a5bb-414f-b55a-23b0130f9115" (UID: "9db214e1-a5bb-414f-b55a-23b0130f9115"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:50:49.966468 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.966441 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-envoy" pod "9db214e1-a5bb-414f-b55a-23b0130f9115" (UID: "9db214e1-a5bb-414f-b55a-23b0130f9115"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:50:49.966743 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:49.966687 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-podinfo" pod "9db214e1-a5bb-414f-b55a-23b0130f9115" (UID: "9db214e1-a5bb-414f-b55a-23b0130f9115"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:50:50.060310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060264 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060310 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060303 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060320 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060336 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060352 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060368 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v92bw\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-kube-api-access-v92bw\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060381 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db214e1-a5bb-414f-b55a-23b0130f9115-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060395 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db214e1-a5bb-414f-b55a-23b0130f9115-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060407 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060425 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db214e1-a5bb-414f-b55a-23b0130f9115-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060435 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87x8b\" (UniqueName: \"kubernetes.io/projected/9db214e1-a5bb-414f-b55a-23b0130f9115-kube-api-access-87x8b\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.060583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.060443 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:50:50.343324 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.343290 4399 generic.go:356] "Generic (PLEG): container finished" podID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerID="cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc" exitCode=0 Oct 31 15:50:50.343463 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.343377 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" Oct 31 15:50:50.343463 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.343403 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" event={"ID":"9db214e1-a5bb-414f-b55a-23b0130f9115","Type":"ContainerDied","Data":"cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc"} Oct 31 15:50:50.343585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.343510 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv" event={"ID":"9db214e1-a5bb-414f-b55a-23b0130f9115","Type":"ContainerDied","Data":"bd5fb6240ec921dc3abd4ab1c32c04214eaafe9231c56b282d68471f1d9ff3a3"} Oct 31 15:50:50.343585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.343543 4399 scope.go:122] "RemoveContainer" containerID="cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc" Oct 31 15:50:50.345375 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.345342 4399 generic.go:356] "Generic (PLEG): container finished" podID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerID="aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe" exitCode=0 Oct 31 15:50:50.345684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.345494 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" event={"ID":"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d","Type":"ContainerDied","Data":"aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe"} Oct 31 15:50:50.345684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.345539 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" event={"ID":"35cad5cd-ecaf-4e90-9f88-9938f5c5d98d","Type":"ContainerDied","Data":"3448ff4251fb41d0ca7569a40f88166078690a47c557ae2cec2743310995aa3c"} Oct 31 15:50:50.345684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.345575 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2" Oct 31 15:50:50.351862 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.351836 4399 generic.go:356] "Generic (PLEG): container finished" podID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerID="c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd" exitCode=0 Oct 31 15:50:50.351959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.351880 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" Oct 31 15:50:50.351959 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.351915 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" event={"ID":"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15","Type":"ContainerDied","Data":"c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd"} Oct 31 15:50:50.352028 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.351960 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk" event={"ID":"e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15","Type":"ContainerDied","Data":"d458befabb9f74e82dfa40332ec1983fc85438ebc741792021a76b8647e61c7e"} Oct 31 15:50:50.355200 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.355178 4399 generic.go:356] "Generic (PLEG): container finished" podID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerID="c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127" exitCode=0 Oct 31 15:50:50.355318 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.355233 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" event={"ID":"ba41a32a-6505-47b2-9fb7-842b885e8369","Type":"ContainerDied","Data":"c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127"} Oct 31 15:50:50.355318 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.355244 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" Oct 31 15:50:50.355318 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.355260 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl" event={"ID":"ba41a32a-6505-47b2-9fb7-842b885e8369","Type":"ContainerDied","Data":"3eb3e2f71e08a7ab5a7d55eae69e04b8aff50d6010e3f9d6edf13385bbe4eec3"} Oct 31 15:50:50.371433 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.371413 4399 scope.go:122] "RemoveContainer" containerID="cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc" Oct 31 15:50:50.371814 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:50.371788 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc\": container with ID starting with cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc not found: ID does not exist" containerID="cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc" Oct 31 15:50:50.371892 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.371824 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc"} err="failed to get container status \"cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc\": rpc error: code = NotFound desc = could not find container \"cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc\": container with ID starting with cb4e09a2bbc92d6ecf585b5f98d496b7e51d180bcf867d2e82ae362cb36c6acc not found: ID does not exist" Oct 31 15:50:50.371892 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.371841 4399 scope.go:122] "RemoveContainer" containerID="aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe" Oct 31 15:50:50.382212 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.382195 4399 scope.go:122] "RemoveContainer" containerID="aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe" Oct 31 15:50:50.382486 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:50.382468 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe\": container with ID starting with aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe not found: ID does not exist" containerID="aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe" Oct 31 15:50:50.382553 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.382495 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe"} err="failed to get container status \"aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe\": rpc error: code = NotFound desc = could not find container \"aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe\": container with ID starting with aa15e988333ed6f2fc049d16f94c1942391c99c3924e276526e1370579f06dfe not found: ID does not exist" Oct 31 15:50:50.382553 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.382512 4399 scope.go:122] "RemoveContainer" containerID="c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd" Oct 31 15:50:50.396586 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.396568 4399 scope.go:122] "RemoveContainer" containerID="c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd" Oct 31 15:50:50.396914 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:50.396887 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd\": container with ID starting with c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd not found: ID does not exist" containerID="c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd" Oct 31 15:50:50.396982 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.396926 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd"} err="failed to get container status \"c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd\": rpc error: code = NotFound desc = could not find container \"c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd\": container with ID starting with c665a64777f39dcb2aad11b8d844656983b714e7f14278fec15ba62b37e6b0fd not found: ID does not exist" Oct 31 15:50:50.396982 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.396947 4399 scope.go:122] "RemoveContainer" containerID="c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127" Oct 31 15:50:50.405753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.405737 4399 scope.go:122] "RemoveContainer" containerID="c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127" Oct 31 15:50:50.405998 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:50.405979 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127\": container with ID starting with c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127 not found: ID does not exist" containerID="c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127" Oct 31 15:50:50.406046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.406004 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127"} err="failed to get container status \"c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127\": rpc error: code = NotFound desc = could not find container \"c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127\": container with ID starting with c0b631f52f3d8c906e3fbc2fd83d15a00bf58131849976d8fa0a6423f28a4127 not found: ID does not exist" Oct 31 15:50:50.508569 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.508544 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2"] Oct 31 15:50:50.511468 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.511451 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-mt2j2"] Oct 31 15:50:50.620221 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.620123 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl"] Oct 31 15:50:50.623630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.623605 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-krbdl"] Oct 31 15:50:50.731888 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.731852 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk"] Oct 31 15:50:50.735933 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.735908 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-cbzzk"] Oct 31 15:50:50.843390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.843362 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv"] Oct 31 15:50:50.850587 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.850562 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-wlhfv"] Oct 31 15:50:50.897364 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.897304 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" path="/var/lib/kubelet/pods/35cad5cd-ecaf-4e90-9f88-9938f5c5d98d/volumes" Oct 31 15:50:50.897814 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.897799 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" path="/var/lib/kubelet/pods/9db214e1-a5bb-414f-b55a-23b0130f9115/volumes" Oct 31 15:50:50.898203 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.898190 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" path="/var/lib/kubelet/pods/ba41a32a-6505-47b2-9fb7-842b885e8369/volumes" Oct 31 15:50:50.898996 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:50:50.898980 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" path="/var/lib/kubelet/pods/e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15/volumes" Oct 31 15:50:56.207062 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:50:56.207015 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:51:12.573163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:12.573085 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz"] Oct 31 15:51:13.375890 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.375854 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62"] Oct 31 15:51:13.376627 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376611 4399 cpu_manager.go:414] "containerMap: removing container" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" Oct 31 15:51:13.376674 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376630 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" Oct 31 15:51:13.376674 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376658 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" Oct 31 15:51:13.376674 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376663 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" Oct 31 15:51:13.376841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376752 4399 cpu_manager.go:414] "containerMap: removing container" podUID="29f054d2-d69a-46eb-957c-d1d46b39bbfe" containerName="maas-api" Oct 31 15:51:13.376841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376759 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29f054d2-d69a-46eb-957c-d1d46b39bbfe" containerName="maas-api" Oct 31 15:51:13.376841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376768 4399 cpu_manager.go:414] "containerMap: removing container" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" Oct 31 15:51:13.376841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376773 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" Oct 31 15:51:13.376841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376795 4399 cpu_manager.go:414] "containerMap: removing container" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" Oct 31 15:51:13.376841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376800 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" Oct 31 15:51:13.377019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376893 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="29f054d2-d69a-46eb-957c-d1d46b39bbfe" containerName="maas-api" Oct 31 15:51:13.377019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376903 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db214e1-a5bb-414f-b55a-23b0130f9115" containerName="istio-proxy" Oct 31 15:51:13.377019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376912 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cad5cd-ecaf-4e90-9f88-9938f5c5d98d" containerName="istio-proxy" Oct 31 15:51:13.377019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376918 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba41a32a-6505-47b2-9fb7-842b885e8369" containerName="istio-proxy" Oct 31 15:51:13.377019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.376927 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f1f8ea-ad57-49d8-b28b-b7a0d1e4dd15" containerName="istio-proxy" Oct 31 15:51:13.380583 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.380568 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.432827 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.432794 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c5ee32-serving-cert\"" Oct 31 15:51:13.436188 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.436162 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62"] Oct 31 15:51:13.553644 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.553612 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-maas-api-tls\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.553826 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.553803 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-metrics-certs\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.553953 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.553940 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s2tw\" (UniqueName: \"kubernetes.io/projected/f7afc771-470c-4206-b9f2-5b97c17c6737-kube-api-access-7s2tw\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.553991 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.553969 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7afc771-470c-4206-b9f2-5b97c17c6737-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.655574 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.655500 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-metrics-certs\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.656066 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.655611 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s2tw\" (UniqueName: \"kubernetes.io/projected/f7afc771-470c-4206-b9f2-5b97c17c6737-kube-api-access-7s2tw\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.656066 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.655633 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7afc771-470c-4206-b9f2-5b97c17c6737-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.656066 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.655655 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-maas-api-tls\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.656413 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.656387 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7afc771-470c-4206-b9f2-5b97c17c6737-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.658262 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.658240 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-maas-api-tls\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.658340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.658287 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-metrics-certs\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.711811 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.711786 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s2tw\" (UniqueName: \"kubernetes.io/projected/f7afc771-470c-4206-b9f2-5b97c17c6737-kube-api-access-7s2tw\") pod \"maas-api-e2e-inf-b-c5ee32-746b85497-qsf62\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:13.994425 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:13.994389 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:14.400663 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:14.400640 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62"] Oct 31 15:51:14.402841 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:51:14.402811 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7afc771_470c_4206_b9f2_5b97c17c6737.slice/crio-4e1cfe21864e360727a79d7e82bc8b87d4a2164de47ec66d2c4f0620ce776a44 WatchSource:0}: Error finding container 4e1cfe21864e360727a79d7e82bc8b87d4a2164de47ec66d2c4f0620ce776a44: Status 404 returned error can't find the container with id 4e1cfe21864e360727a79d7e82bc8b87d4a2164de47ec66d2c4f0620ce776a44 Oct 31 15:51:14.508530 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:14.508488 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" event={"ID":"f7afc771-470c-4206-b9f2-5b97c17c6737","Type":"ContainerStarted","Data":"4e1cfe21864e360727a79d7e82bc8b87d4a2164de47ec66d2c4f0620ce776a44"} Oct 31 15:51:15.515372 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:15.515330 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" event={"ID":"f7afc771-470c-4206-b9f2-5b97c17c6737","Type":"ContainerStarted","Data":"c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc"} Oct 31 15:51:15.515889 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:15.515394 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:15.629756 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:15.629613 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" podStartSLOduration=1.765242781 podStartE2EDuration="2.629598012s" totalImagesPullingTime="864.355231ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:51:15.628495556 +0000 UTC m=+3918.839736968" watchObservedRunningTime="2026-10-31 15:51:15.629598012 +0000 UTC m=+3918.840839424" Oct 31 15:51:17.573993 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:17.573956 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" podUID="363d1acf-1182-4f18-855a-8b8b8fca41ed" containerName="payload-processing" containerID="cri-o://2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4" gracePeriod=25 Oct 31 15:51:17.845581 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:17.845559 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:51:17.915146 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:17.915118 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363d1acf-1182-4f18-855a-8b8b8fca41ed-kube-api-access-rmdrz\" (UniqueName: \"kubernetes.io/projected/363d1acf-1182-4f18-855a-8b8b8fca41ed-kube-api-access-rmdrz\") pod \"363d1acf-1182-4f18-855a-8b8b8fca41ed\" (UID: \"363d1acf-1182-4f18-855a-8b8b8fca41ed\") " Oct 31 15:51:17.915301 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:17.915184 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363d1acf-1182-4f18-855a-8b8b8fca41ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363d1acf-1182-4f18-855a-8b8b8fca41ed-plugins-config-volume\") pod \"363d1acf-1182-4f18-855a-8b8b8fca41ed\" (UID: \"363d1acf-1182-4f18-855a-8b8b8fca41ed\") " Oct 31 15:51:17.915581 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:17.915543 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363d1acf-1182-4f18-855a-8b8b8fca41ed-plugins-config-volume" pod "363d1acf-1182-4f18-855a-8b8b8fca41ed" (UID: "363d1acf-1182-4f18-855a-8b8b8fca41ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:51:17.917480 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:17.917454 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363d1acf-1182-4f18-855a-8b8b8fca41ed-kube-api-access-rmdrz" pod "363d1acf-1182-4f18-855a-8b8b8fca41ed" (UID: "363d1acf-1182-4f18-855a-8b8b8fca41ed"). InnerVolumeSpecName "kube-api-access-rmdrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:51:18.016452 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.016408 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmdrz\" (UniqueName: \"kubernetes.io/projected/363d1acf-1182-4f18-855a-8b8b8fca41ed-kube-api-access-rmdrz\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:18.016452 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.016446 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/363d1acf-1182-4f18-855a-8b8b8fca41ed-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:18.401340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.401304 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f"] Oct 31 15:51:18.506732 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.506676 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh"] Oct 31 15:51:18.535860 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.535829 4399 generic.go:356] "Generic (PLEG): container finished" podID="363d1acf-1182-4f18-855a-8b8b8fca41ed" containerID="2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4" exitCode=0 Oct 31 15:51:18.536067 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.535884 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" Oct 31 15:51:18.536067 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.535924 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" event={"ID":"363d1acf-1182-4f18-855a-8b8b8fca41ed","Type":"ContainerDied","Data":"2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4"} Oct 31 15:51:18.536067 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.535991 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz" event={"ID":"363d1acf-1182-4f18-855a-8b8b8fca41ed","Type":"ContainerDied","Data":"224fb002dc6eadf532e65b171121f58c2e7ae2a7db853b12b471b783d7022476"} Oct 31 15:51:18.536067 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.536019 4399 scope.go:122] "RemoveContainer" containerID="2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4" Oct 31 15:51:18.546186 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.546170 4399 scope.go:122] "RemoveContainer" containerID="2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4" Oct 31 15:51:18.546465 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:51:18.546444 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4\": container with ID starting with 2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4 not found: ID does not exist" containerID="2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4" Oct 31 15:51:18.546509 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.546474 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4"} err="failed to get container status \"2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4\": rpc error: code = NotFound desc = could not find container \"2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4\": container with ID starting with 2b1c0fc35c250b5505b818cb04bd5982e8b96eb5a7579239870cd7d53562c1c4 not found: ID does not exist" Oct 31 15:51:18.753194 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.753158 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz"] Oct 31 15:51:18.755877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.755857 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-c4840c-7f5bc8988f-zqtcz"] Oct 31 15:51:18.898234 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:18.898203 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363d1acf-1182-4f18-855a-8b8b8fca41ed" path="/var/lib/kubelet/pods/363d1acf-1182-4f18-855a-8b8b8fca41ed/volumes" Oct 31 15:51:21.527421 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:21.527391 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:51:23.402481 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.402424 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" podUID="0bb0f96e-61db-416c-8486-b2d61faac230" containerName="payload-processing" containerID="cri-o://a69c6b17dba4db1a89d2673cea38a5b91737343cd43e2305ef662d075840ff65" gracePeriod=25 Oct 31 15:51:23.507684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.507627 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" podUID="30e7eb52-dc71-45e9-8938-d1eefc8f0526" containerName="payload-pre-processing" containerID="cri-o://b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9" gracePeriod=25 Oct 31 15:51:23.572307 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.572268 4399 generic.go:356] "Generic (PLEG): container finished" podID="0bb0f96e-61db-416c-8486-b2d61faac230" containerID="a69c6b17dba4db1a89d2673cea38a5b91737343cd43e2305ef662d075840ff65" exitCode=0 Oct 31 15:51:23.572643 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.572340 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" event={"ID":"0bb0f96e-61db-416c-8486-b2d61faac230","Type":"ContainerDied","Data":"a69c6b17dba4db1a89d2673cea38a5b91737343cd43e2305ef662d075840ff65"} Oct 31 15:51:23.708108 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.708078 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:51:23.778535 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.778512 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:51:23.820094 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.820061 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bb0f96e-61db-416c-8486-b2d61faac230-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb0f96e-61db-416c-8486-b2d61faac230-plugins-config-volume\") pod \"0bb0f96e-61db-416c-8486-b2d61faac230\" (UID: \"0bb0f96e-61db-416c-8486-b2d61faac230\") " Oct 31 15:51:23.820301 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.820108 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bb0f96e-61db-416c-8486-b2d61faac230-kube-api-access-ljq2p\" (UniqueName: \"kubernetes.io/projected/0bb0f96e-61db-416c-8486-b2d61faac230-kube-api-access-ljq2p\") pod \"0bb0f96e-61db-416c-8486-b2d61faac230\" (UID: \"0bb0f96e-61db-416c-8486-b2d61faac230\") " Oct 31 15:51:23.820787 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.820752 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bb0f96e-61db-416c-8486-b2d61faac230-plugins-config-volume" pod "0bb0f96e-61db-416c-8486-b2d61faac230" (UID: "0bb0f96e-61db-416c-8486-b2d61faac230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:51:23.823189 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.823159 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bb0f96e-61db-416c-8486-b2d61faac230-kube-api-access-ljq2p" pod "0bb0f96e-61db-416c-8486-b2d61faac230" (UID: "0bb0f96e-61db-416c-8486-b2d61faac230"). InnerVolumeSpecName "kube-api-access-ljq2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:51:23.921128 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.921095 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30e7eb52-dc71-45e9-8938-d1eefc8f0526-kube-api-access-9pxrn\" (UniqueName: \"kubernetes.io/projected/30e7eb52-dc71-45e9-8938-d1eefc8f0526-kube-api-access-9pxrn\") pod \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\" (UID: \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\") " Oct 31 15:51:23.921316 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.921182 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30e7eb52-dc71-45e9-8938-d1eefc8f0526-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e7eb52-dc71-45e9-8938-d1eefc8f0526-plugins-config-volume\") pod \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\" (UID: \"30e7eb52-dc71-45e9-8938-d1eefc8f0526\") " Oct 31 15:51:23.921530 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.921514 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bb0f96e-61db-416c-8486-b2d61faac230-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:23.921598 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.921532 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljq2p\" (UniqueName: \"kubernetes.io/projected/0bb0f96e-61db-416c-8486-b2d61faac230-kube-api-access-ljq2p\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:23.921598 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.921514 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30e7eb52-dc71-45e9-8938-d1eefc8f0526-plugins-config-volume" pod "30e7eb52-dc71-45e9-8938-d1eefc8f0526" (UID: "30e7eb52-dc71-45e9-8938-d1eefc8f0526"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:51:23.923411 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:23.923389 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30e7eb52-dc71-45e9-8938-d1eefc8f0526-kube-api-access-9pxrn" pod "30e7eb52-dc71-45e9-8938-d1eefc8f0526" (UID: "30e7eb52-dc71-45e9-8938-d1eefc8f0526"). InnerVolumeSpecName "kube-api-access-9pxrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:51:24.022859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.022826 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pxrn\" (UniqueName: \"kubernetes.io/projected/30e7eb52-dc71-45e9-8938-d1eefc8f0526-kube-api-access-9pxrn\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:24.022859 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.022853 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30e7eb52-dc71-45e9-8938-d1eefc8f0526-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:51:24.580675 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.580641 4399 generic.go:356] "Generic (PLEG): container finished" podID="30e7eb52-dc71-45e9-8938-d1eefc8f0526" containerID="b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9" exitCode=0 Oct 31 15:51:24.581140 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.580686 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" Oct 31 15:51:24.581140 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.580732 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" event={"ID":"30e7eb52-dc71-45e9-8938-d1eefc8f0526","Type":"ContainerDied","Data":"b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9"} Oct 31 15:51:24.581140 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.580800 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh" event={"ID":"30e7eb52-dc71-45e9-8938-d1eefc8f0526","Type":"ContainerDied","Data":"2e3b60a8dbe440e6f42a394dd554632ac92d208cc3e55b2254346d326ea374de"} Oct 31 15:51:24.581140 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.580840 4399 scope.go:122] "RemoveContainer" containerID="b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9" Oct 31 15:51:24.582447 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.582429 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" Oct 31 15:51:24.582447 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.582437 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f" event={"ID":"0bb0f96e-61db-416c-8486-b2d61faac230","Type":"ContainerDied","Data":"f66ae51b69043bd5553130f7939dceb2d3bbac058b6a4a52b9f851e1c05d22d1"} Oct 31 15:51:24.592440 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.592360 4399 scope.go:122] "RemoveContainer" containerID="b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9" Oct 31 15:51:24.592671 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:51:24.592650 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9\": container with ID starting with b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9 not found: ID does not exist" containerID="b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9" Oct 31 15:51:24.592785 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.592680 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9"} err="failed to get container status \"b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9\": rpc error: code = NotFound desc = could not find container \"b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9\": container with ID starting with b3048a16c27a071b1882e1df377170507864b319c764e0dc87e43425296917a9 not found: ID does not exist" Oct 31 15:51:24.592785 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.592713 4399 scope.go:122] "RemoveContainer" containerID="a69c6b17dba4db1a89d2673cea38a5b91737343cd43e2305ef662d075840ff65" Oct 31 15:51:24.742656 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.742627 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f"] Oct 31 15:51:24.791962 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.791933 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b18df8-5b958bbc4c-c5m7f"] Oct 31 15:51:24.853180 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.853149 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh"] Oct 31 15:51:24.858943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.858920 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b18df8-867b94fd87-xn9nh"] Oct 31 15:51:24.897689 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.897660 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bb0f96e-61db-416c-8486-b2d61faac230" path="/var/lib/kubelet/pods/0bb0f96e-61db-416c-8486-b2d61faac230/volumes" Oct 31 15:51:24.898070 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:24.898055 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30e7eb52-dc71-45e9-8938-d1eefc8f0526" path="/var/lib/kubelet/pods/30e7eb52-dc71-45e9-8938-d1eefc8f0526/volumes" Oct 31 15:51:26.205819 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:51:26.205786 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:51:40.718069 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:40.718034 4399 generic.go:356] "Generic (PLEG): container finished" podID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerID="e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1" exitCode=0 Oct 31 15:51:40.718510 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:40.718114 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" event={"ID":"f83e1c67-842d-4759-8e56-7d14d56f4e51","Type":"ContainerDied","Data":"e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1"} Oct 31 15:51:40.718577 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:40.718556 4399 scope.go:122] "RemoveContainer" containerID="e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1" Oct 31 15:51:40.720322 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:40.720295 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:40.720400 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:40.720364 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:40.720400 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:40.720397 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:41.726204 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:41.726165 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" event={"ID":"f83e1c67-842d-4759-8e56-7d14d56f4e51","Type":"ContainerStarted","Data":"88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84"} Oct 31 15:51:42.653725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:42.646906 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:51:42.653725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:42.647328 4399 patch_prober.go:29] interesting pod/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 15:51:42.653725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:42.647375 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 15:51:43.665916 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:43.665813 4399 patch_prober.go:29] interesting pod/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 15:51:43.665916 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:43.665884 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 15:51:44.655737 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.655176 4399 patch_prober.go:29] interesting pod/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 15:51:44.655737 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.655241 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 15:51:44.773025 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.772992 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8"] Oct 31 15:51:44.773867 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.773845 4399 cpu_manager.go:414] "containerMap: removing container" podUID="363d1acf-1182-4f18-855a-8b8b8fca41ed" containerName="payload-processing" Oct 31 15:51:44.773994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.773867 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363d1acf-1182-4f18-855a-8b8b8fca41ed" containerName="payload-processing" Oct 31 15:51:44.773994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.773923 4399 cpu_manager.go:414] "containerMap: removing container" podUID="0bb0f96e-61db-416c-8486-b2d61faac230" containerName="payload-processing" Oct 31 15:51:44.773994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.773931 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bb0f96e-61db-416c-8486-b2d61faac230" containerName="payload-processing" Oct 31 15:51:44.773994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.773946 4399 cpu_manager.go:414] "containerMap: removing container" podUID="30e7eb52-dc71-45e9-8938-d1eefc8f0526" containerName="payload-pre-processing" Oct 31 15:51:44.773994 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.773955 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30e7eb52-dc71-45e9-8938-d1eefc8f0526" containerName="payload-pre-processing" Oct 31 15:51:44.774252 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.774088 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="363d1acf-1182-4f18-855a-8b8b8fca41ed" containerName="payload-processing" Oct 31 15:51:44.774252 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.774113 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bb0f96e-61db-416c-8486-b2d61faac230" containerName="payload-processing" Oct 31 15:51:44.774252 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.774126 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="30e7eb52-dc71-45e9-8938-d1eefc8f0526" containerName="payload-pre-processing" Oct 31 15:51:44.777858 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.777836 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.809147 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.809116 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.809274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.809190 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.809319 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.809282 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdzg\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-kube-api-access-skdzg\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.809406 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.809387 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.809485 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.809421 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.809558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.809539 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.809621 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.809601 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.809666 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.809642 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.809735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.809662 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.822476 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.822443 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm"] Oct 31 15:51:44.826521 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.826503 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff"] Oct 31 15:51:44.826678 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.826657 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.830924 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.830905 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.867722 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.867080 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" Oct 31 15:51:44.875545 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.875498 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8"] Oct 31 15:51:44.911567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.911480 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.911567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.911528 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.911567 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.911546 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.911889 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.911629 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.911889 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.911692 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.911889 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.911876 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5wjx\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-kube-api-access-r5wjx\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.912065 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.911928 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.912065 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.911959 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw54c\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-kube-api-access-qw54c\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.912065 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.911987 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.912065 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912015 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.912274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912082 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.912274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912102 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.912274 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912107 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.912529 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912508 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.912609 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912554 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdzg\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-kube-api-access-skdzg\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.912609 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912591 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.912758 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912659 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.912758 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912683 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.912758 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912743 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.912929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912782 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.912929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912840 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.912929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912892 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.912929 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912916 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.913149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.912974 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ec614-34ae-494b-8652-d1865243f5b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.913149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.913003 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:44.913149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.913042 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ec614-34ae-494b-8652-d1865243f5b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:44.913149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.913093 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.913149 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.913116 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.913590 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.913435 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.913680 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.913641 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.914738 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.914707 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.914876 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.914859 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.915214 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.915194 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.915377 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.915361 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.967727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.949766 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff"] Oct 31 15:51:44.973502 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.973465 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm"] Oct 31 15:51:44.973874 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.973848 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:44.975311 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:44.975289 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdzg\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-kube-api-access-skdzg\") pod \"maas-default-gateway-openshift-default-67bf986f99-97lt8\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:45.014161 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014124 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ec614-34ae-494b-8652-d1865243f5b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.014161 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014160 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.014414 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014187 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ec614-34ae-494b-8652-d1865243f5b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.014414 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014382 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.014534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014449 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.014534 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014495 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.014650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014544 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5wjx\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-kube-api-access-r5wjx\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.014650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014588 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.014650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014615 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw54c\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-kube-api-access-qw54c\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.014650 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014643 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.014893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014747 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.014893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014773 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.014893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014835 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.014893 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014884 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.015120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014910 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.015120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014945 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.015120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014964 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.015120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.014982 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.015120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.015024 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.015388 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.015176 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.017361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.015984 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.017361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.016116 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ec614-34ae-494b-8652-d1865243f5b5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.017361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.016606 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.017361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.016784 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.017361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.016894 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.017361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.016920 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.017361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.017145 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.017361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.017158 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.017361 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.017323 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.018523 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.018496 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ec614-34ae-494b-8652-d1865243f5b5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.018735 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.018686 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.020043 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.020012 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.071611 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.071572 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.071813 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.071778 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5wjx\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-kube-api-access-r5wjx\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.071921 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.071900 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw54c\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-kube-api-access-qw54c\") pod \"maas-default-gateway-openshift-default-67bf986f99-gvqff\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.072148 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.072125 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pkkhm\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.091627 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.091607 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:45.170440 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.168496 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:45.174045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.173982 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:45.565841 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.562712 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8"] Oct 31 15:51:45.566527 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:51:45.566493 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9ff191_9669_4e72_b06d_6b1518c1dea1.slice/crio-0f4bf0fea12058b3b37ab9cfd372012562ac3b0911be181d04ce1ea99fe529ed WatchSource:0}: Error finding container 0f4bf0fea12058b3b37ab9cfd372012562ac3b0911be181d04ce1ea99fe529ed: Status 404 returned error can't find the container with id 0f4bf0fea12058b3b37ab9cfd372012562ac3b0911be181d04ce1ea99fe529ed Oct 31 15:51:45.569778 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.569747 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:45.569863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.569815 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:45.569863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.569846 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:45.596684 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.595995 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm"] Oct 31 15:51:45.598174 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:51:45.598153 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2c4a7a_c75b_40cc_b6bb_2af983ef67da.slice/crio-9ff9e4a68865e77ee5491f1195e0e1f1772d0d83e3fcda5a58ea2c4e0f0e5278 WatchSource:0}: Error finding container 9ff9e4a68865e77ee5491f1195e0e1f1772d0d83e3fcda5a58ea2c4e0f0e5278: Status 404 returned error can't find the container with id 9ff9e4a68865e77ee5491f1195e0e1f1772d0d83e3fcda5a58ea2c4e0f0e5278 Oct 31 15:51:45.601019 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.600988 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:45.601125 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.601066 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:45.601125 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.601110 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:45.652385 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.652267 4399 patch_prober.go:29] interesting pod/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 15:51:45.652385 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.652323 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 15:51:45.671715 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.671591 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff"] Oct 31 15:51:45.679485 ip-10-0-137-160 kubenswrapper[4399]: W1031 15:51:45.679431 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod148ec614_34ae_494b_8652_d1865243f5b5.slice/crio-f7bb61d0e8f4640378543cf61a8f36c593095f04618f975cbb9c6bd73cf0b1f4 WatchSource:0}: Error finding container f7bb61d0e8f4640378543cf61a8f36c593095f04618f975cbb9c6bd73cf0b1f4: Status 404 returned error can't find the container with id f7bb61d0e8f4640378543cf61a8f36c593095f04618f975cbb9c6bd73cf0b1f4 Oct 31 15:51:45.684312 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.682997 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:45.684312 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.683081 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:45.684312 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.683121 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 15:51:45.783482 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.783448 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" event={"ID":"3c9ff191-9669-4e72-b06d-6b1518c1dea1","Type":"ContainerStarted","Data":"3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f"} Oct 31 15:51:45.784295 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.784127 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" event={"ID":"3c9ff191-9669-4e72-b06d-6b1518c1dea1","Type":"ContainerStarted","Data":"0f4bf0fea12058b3b37ab9cfd372012562ac3b0911be181d04ce1ea99fe529ed"} Oct 31 15:51:45.786002 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.785976 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" event={"ID":"148ec614-34ae-494b-8652-d1865243f5b5","Type":"ContainerStarted","Data":"f7bb61d0e8f4640378543cf61a8f36c593095f04618f975cbb9c6bd73cf0b1f4"} Oct 31 15:51:45.789073 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.788283 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" event={"ID":"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da","Type":"ContainerStarted","Data":"e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac"} Oct 31 15:51:45.789073 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.788331 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" event={"ID":"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da","Type":"ContainerStarted","Data":"9ff9e4a68865e77ee5491f1195e0e1f1772d0d83e3fcda5a58ea2c4e0f0e5278"} Oct 31 15:51:45.906895 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:45.905431 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podStartSLOduration=1.905410756 podStartE2EDuration="1.905410756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:51:45.89982436 +0000 UTC m=+3949.111065773" watchObservedRunningTime="2026-10-31 15:51:45.905410756 +0000 UTC m=+3949.116652168" Oct 31 15:51:46.092086 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.092052 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:46.093998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.093962 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-97lt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 15:51:46.094152 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.094016 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 15:51:46.176436 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.176336 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:46.178781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.176848 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pkkhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 15:51:46.178781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.176906 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 15:51:46.645212 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.645172 4399 patch_prober.go:29] interesting pod/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 31 15:51:46.645518 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.645483 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 31 15:51:46.828322 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.826111 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" event={"ID":"148ec614-34ae-494b-8652-d1865243f5b5","Type":"ContainerStarted","Data":"2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a"} Oct 31 15:51:46.960531 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:46.960014 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podStartSLOduration=2.959991004 podStartE2EDuration="2.959991004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:51:46.028346363 +0000 UTC m=+3949.239587775" watchObservedRunningTime="2026-10-31 15:51:46.959991004 +0000 UTC m=+3950.171232418" Oct 31 15:51:47.092723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.092148 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-97lt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 15:51:47.092723 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.092245 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 15:51:47.186654 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.183788 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:47.186654 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.185181 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pkkhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 15:51:47.186654 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.185260 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 15:51:47.187106 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.187082 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:51:47.193727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.190803 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 15:51:47.876725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.876121 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:51:47.885734 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.877977 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:51:47.889979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:47.889950 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:51:48.004585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:48.004523 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podStartSLOduration=4.004501507 podStartE2EDuration="4.004501507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 15:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 15:51:46.962056232 +0000 UTC m=+3950.173297644" watchObservedRunningTime="2026-10-31 15:51:48.004501507 +0000 UTC m=+3951.215742922" Oct 31 15:51:48.102725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:48.098890 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-97lt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 15:51:48.102725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:48.098945 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 15:51:48.171475 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:48.171391 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pkkhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 15:51:48.171781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:48.171753 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 15:51:48.176125 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:48.176097 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:51:48.176329 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:48.176301 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 15:51:49.092168 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:49.092130 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-97lt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 15:51:49.092839 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:49.092806 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 15:51:49.174138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:49.174103 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pkkhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 15:51:49.174810 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:49.174777 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 15:51:49.175000 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:49.174679 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:51:49.175125 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:49.175107 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 15:51:50.101724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:50.093125 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-97lt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 15:51:50.101724 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:50.093187 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 15:51:50.174120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:50.174082 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pkkhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 15:51:50.174401 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:50.174372 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 15:51:50.174849 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:50.174828 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:51:50.175017 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:50.174996 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 15:51:51.095160 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:51.095119 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-97lt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 15:51:51.095477 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:51.095444 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 15:51:51.171453 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:51.171059 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pkkhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 15:51:51.171453 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:51.171112 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 15:51:51.195721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:51.175027 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:51:51.195721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:51.175078 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 15:51:52.121979 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:52.121929 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-97lt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 15:51:52.122271 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:52.122228 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 15:51:52.201726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:52.187051 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pkkhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 15:51:52.201726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:52.187278 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 15:51:52.201726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:52.187888 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:51:52.201726 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:52.187921 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 15:51:53.092317 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:53.092285 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-97lt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 15:51:53.092543 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:53.092341 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 15:51:53.171402 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:53.171370 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pkkhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 15:51:53.171547 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:53.171446 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 15:51:53.174840 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:53.174813 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:51:53.174943 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:53.174862 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 15:51:54.093039 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:54.092999 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-97lt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 31 15:51:54.093449 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:54.093067 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 31 15:51:54.170182 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:54.170141 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pkkhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 31 15:51:54.170357 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:54.170212 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 31 15:51:54.174570 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:54.174541 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" start-of-body= Oct 31 15:51:54.174725 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:54.174597 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": dial tcp 10.133.0.159:15021: connect: connection refused" Oct 31 15:51:56.031558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.031525 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:56.033397 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.033376 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:56.034094 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.034073 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 15:51:56.129954 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.129919 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:56.132193 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.132167 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:56.133807 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.133788 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 15:51:56.208527 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:51:56.208487 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:51:56.214874 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.214839 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 15:51:56.215014 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.214899 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.159:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 15:51:56.363558 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.363472 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:56.363922 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.363894 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:56.364822 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:56.364800 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 15:51:58.131275 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:51:58.131237 4399 scope.go:122] "RemoveContainer" containerID="5a770388fe668cc73dd227c2b5d0e15f273b30f71d39acb7e3b676e729de8e40" Oct 31 15:52:26.208472 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:52:26.208426 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:52:34.301585 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.301550 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt"] Oct 31 15:52:34.302200 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.301767 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" podUID="fd35a41e-27bd-4139-99fa-60f81f538369" containerName="maas-api" containerID="cri-o://d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8" gracePeriod=30 Oct 31 15:52:34.631901 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.631879 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:52:34.711378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.710055 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd35a41e-27bd-4139-99fa-60f81f538369-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd35a41e-27bd-4139-99fa-60f81f538369-openshift-service-ca\") pod \"fd35a41e-27bd-4139-99fa-60f81f538369\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " Oct 31 15:52:34.711378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.710155 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-metrics-certs\") pod \"fd35a41e-27bd-4139-99fa-60f81f538369\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " Oct 31 15:52:34.711378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.710309 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd35a41e-27bd-4139-99fa-60f81f538369-kube-api-access-4ktfx\" (UniqueName: \"kubernetes.io/projected/fd35a41e-27bd-4139-99fa-60f81f538369-kube-api-access-4ktfx\") pod \"fd35a41e-27bd-4139-99fa-60f81f538369\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " Oct 31 15:52:34.711378 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.710361 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls\") pod \"fd35a41e-27bd-4139-99fa-60f81f538369\" (UID: \"fd35a41e-27bd-4139-99fa-60f81f538369\") " Oct 31 15:52:34.715539 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.714429 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd35a41e-27bd-4139-99fa-60f81f538369-openshift-service-ca" pod "fd35a41e-27bd-4139-99fa-60f81f538369" (UID: "fd35a41e-27bd-4139-99fa-60f81f538369"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:52:34.722079 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.722012 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-metrics-certs" pod "fd35a41e-27bd-4139-99fa-60f81f538369" (UID: "fd35a41e-27bd-4139-99fa-60f81f538369"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:52:34.722235 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.722073 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls" pod "fd35a41e-27bd-4139-99fa-60f81f538369" (UID: "fd35a41e-27bd-4139-99fa-60f81f538369"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:52:34.728342 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.728283 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd35a41e-27bd-4139-99fa-60f81f538369-kube-api-access-4ktfx" pod "fd35a41e-27bd-4139-99fa-60f81f538369" (UID: "fd35a41e-27bd-4139-99fa-60f81f538369"). InnerVolumeSpecName "kube-api-access-4ktfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:52:34.812709 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.812662 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:34.812870 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.812721 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ktfx\" (UniqueName: \"kubernetes.io/projected/fd35a41e-27bd-4139-99fa-60f81f538369-kube-api-access-4ktfx\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:34.812870 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.812741 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fd35a41e-27bd-4139-99fa-60f81f538369-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:34.812870 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:34.812755 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd35a41e-27bd-4139-99fa-60f81f538369-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:52:35.303063 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:35.303026 4399 generic.go:356] "Generic (PLEG): container finished" podID="fd35a41e-27bd-4139-99fa-60f81f538369" containerID="d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8" exitCode=0 Oct 31 15:52:35.303679 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:35.303652 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" event={"ID":"fd35a41e-27bd-4139-99fa-60f81f538369","Type":"ContainerDied","Data":"d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8"} Oct 31 15:52:35.303879 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:35.303851 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" event={"ID":"fd35a41e-27bd-4139-99fa-60f81f538369","Type":"ContainerDied","Data":"88301e6ccbb78889c22e09aa7181d7c2b291410050b883d879686043a6a04fa5"} Oct 31 15:52:35.304015 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:35.303904 4399 scope.go:122] "RemoveContainer" containerID="d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8" Oct 31 15:52:35.304097 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:35.304081 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt" Oct 31 15:52:35.327971 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:35.327941 4399 scope.go:122] "RemoveContainer" containerID="d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8" Oct 31 15:52:35.328325 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:52:35.328300 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8\": container with ID starting with d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8 not found: ID does not exist" containerID="d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8" Oct 31 15:52:35.328384 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:35.328336 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8"} err="failed to get container status \"d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8\": rpc error: code = NotFound desc = could not find container \"d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8\": container with ID starting with d6fb00a0a6f3118349dbe4e28301cd7b3d9b32ff1f04e4ed03396888366f23b8 not found: ID does not exist" Oct 31 15:52:35.471511 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:35.471478 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt"] Oct 31 15:52:35.521266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:35.521221 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b18df8-fdbc89594-cjfvt"] Oct 31 15:52:36.897356 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:52:36.897311 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd35a41e-27bd-4139-99fa-60f81f538369" path="/var/lib/kubelet/pods/fd35a41e-27bd-4139-99fa-60f81f538369/volumes" Oct 31 15:52:56.209586 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:52:56.209543 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:53:26.206800 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:53:26.206766 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:53:56.208038 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:53:56.208000 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:54:26.206378 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:54:26.206342 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:54:47.081138 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:47.081103 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49"] Oct 31 15:54:47.081778 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:47.081353 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" containerID="cri-o://c6ebc117271d9caf50d4fe282a325148c26a7da45fee741a2863c276664220b3" gracePeriod=30 Oct 31 15:54:52.283025 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.282896 4399 generic.go:356] "Generic (PLEG): container finished" podID="05f4b118-d329-4758-99d6-87367efd0468" containerID="c6ebc117271d9caf50d4fe282a325148c26a7da45fee741a2863c276664220b3" exitCode=0 Oct 31 15:54:52.283781 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.283746 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" event={"ID":"05f4b118-d329-4758-99d6-87367efd0468","Type":"ContainerDied","Data":"c6ebc117271d9caf50d4fe282a325148c26a7da45fee741a2863c276664220b3"} Oct 31 15:54:52.283892 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.283857 4399 scope.go:122] "RemoveContainer" containerID="accf8286ee975bb7cab62e4456f7e79c7d7da2cf5eec6027af536eac1e5dd349" Oct 31 15:54:52.349544 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.349491 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:54:52.422178 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422152 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-envoy\") pod \"05f4b118-d329-4758-99d6-87367efd0468\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " Oct 31 15:54:52.422302 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422188 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-kube-api-access-kx46x\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-kube-api-access-kx46x\") pod \"05f4b118-d329-4758-99d6-87367efd0468\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " Oct 31 15:54:52.422302 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422235 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-credential-socket\") pod \"05f4b118-d329-4758-99d6-87367efd0468\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " Oct 31 15:54:52.422302 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422256 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-certs\") pod \"05f4b118-d329-4758-99d6-87367efd0468\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " Oct 31 15:54:52.422458 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422316 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-istio-token\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-istio-token\") pod \"05f4b118-d329-4758-99d6-87367efd0468\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " Oct 31 15:54:52.422458 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422391 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-socket\") pod \"05f4b118-d329-4758-99d6-87367efd0468\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " Oct 31 15:54:52.422577 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422462 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f4b118-d329-4758-99d6-87367efd0468-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f4b118-d329-4758-99d6-87367efd0468-istiod-ca-cert\") pod \"05f4b118-d329-4758-99d6-87367efd0468\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " Oct 31 15:54:52.422577 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422502 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-data\") pod \"05f4b118-d329-4758-99d6-87367efd0468\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " Oct 31 15:54:52.422577 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422553 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-credential-socket" pod "05f4b118-d329-4758-99d6-87367efd0468" (UID: "05f4b118-d329-4758-99d6-87367efd0468"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:52.422577 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422565 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05f4b118-d329-4758-99d6-87367efd0468-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f4b118-d329-4758-99d6-87367efd0468-istio-podinfo\") pod \"05f4b118-d329-4758-99d6-87367efd0468\" (UID: \"05f4b118-d329-4758-99d6-87367efd0468\") " Oct 31 15:54:52.422816 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422579 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-certs" pod "05f4b118-d329-4758-99d6-87367efd0468" (UID: "05f4b118-d329-4758-99d6-87367efd0468"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:52.422816 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422755 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-socket" pod "05f4b118-d329-4758-99d6-87367efd0468" (UID: "05f4b118-d329-4758-99d6-87367efd0468"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:52.422957 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.422923 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-data" pod "05f4b118-d329-4758-99d6-87367efd0468" (UID: "05f4b118-d329-4758-99d6-87367efd0468"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:52.423056 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.423032 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f4b118-d329-4758-99d6-87367efd0468-istiod-ca-cert" pod "05f4b118-d329-4758-99d6-87367efd0468" (UID: "05f4b118-d329-4758-99d6-87367efd0468"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:54:52.423139 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.423122 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:52.423204 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.423141 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:52.423204 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.423150 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:52.423204 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.423159 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:52.424746 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.424672 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-istio-token" pod "05f4b118-d329-4758-99d6-87367efd0468" (UID: "05f4b118-d329-4758-99d6-87367efd0468"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:54:52.424935 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.424912 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05f4b118-d329-4758-99d6-87367efd0468-istio-podinfo" pod "05f4b118-d329-4758-99d6-87367efd0468" (UID: "05f4b118-d329-4758-99d6-87367efd0468"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:54:52.425053 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.425036 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-envoy" pod "05f4b118-d329-4758-99d6-87367efd0468" (UID: "05f4b118-d329-4758-99d6-87367efd0468"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:54:52.425114 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.425077 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-kube-api-access-kx46x" pod "05f4b118-d329-4758-99d6-87367efd0468" (UID: "05f4b118-d329-4758-99d6-87367efd0468"). InnerVolumeSpecName "kube-api-access-kx46x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:54:52.524501 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.524470 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05f4b118-d329-4758-99d6-87367efd0468-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:52.524501 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.524500 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05f4b118-d329-4758-99d6-87367efd0468-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:52.524685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.524513 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05f4b118-d329-4758-99d6-87367efd0468-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:52.524685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.524526 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx46x\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-kube-api-access-kx46x\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:52.524685 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:52.524537 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05f4b118-d329-4758-99d6-87367efd0468-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:53.290640 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:53.290610 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" Oct 31 15:54:53.291125 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:53.290606 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49" event={"ID":"05f4b118-d329-4758-99d6-87367efd0468","Type":"ContainerDied","Data":"f538d0c601a78c091b2eb5fcda259ec8681a87c41b06610c4e8205b0dacf7dea"} Oct 31 15:54:53.291125 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:53.290775 4399 scope.go:122] "RemoveContainer" containerID="c6ebc117271d9caf50d4fe282a325148c26a7da45fee741a2863c276664220b3" Oct 31 15:54:53.504017 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:53.503984 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49"] Oct 31 15:54:53.554032 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:53.553962 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-883697-openshift-default-5dbfd8769c-vfh49"] Oct 31 15:54:54.669588 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:54.669559 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5"] Oct 31 15:54:54.899718 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:54.899666 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f4b118-d329-4758-99d6-87367efd0468" path="/var/lib/kubelet/pods/05f4b118-d329-4758-99d6-87367efd0468/volumes" Oct 31 15:54:55.746965 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:55.746934 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg"] Oct 31 15:54:55.747413 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:55.747145 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" podUID="12a04ce2-d74b-4cfb-87e9-36f916d4f590" containerName="maas-api" containerID="cri-o://08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058" gracePeriod=30 Oct 31 15:54:56.029347 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.029035 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:54:56.067492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.065486 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls\") pod \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " Oct 31 15:54:56.067492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.065568 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-metrics-certs\") pod \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " Oct 31 15:54:56.067492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.065839 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12a04ce2-d74b-4cfb-87e9-36f916d4f590-kube-api-access-nwsvq\" (UniqueName: \"kubernetes.io/projected/12a04ce2-d74b-4cfb-87e9-36f916d4f590-kube-api-access-nwsvq\") pod \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " Oct 31 15:54:56.067492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.065903 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12a04ce2-d74b-4cfb-87e9-36f916d4f590-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a04ce2-d74b-4cfb-87e9-36f916d4f590-openshift-service-ca\") pod \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\" (UID: \"12a04ce2-d74b-4cfb-87e9-36f916d4f590\") " Oct 31 15:54:56.067492 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.067434 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a04ce2-d74b-4cfb-87e9-36f916d4f590-openshift-service-ca" pod "12a04ce2-d74b-4cfb-87e9-36f916d4f590" (UID: "12a04ce2-d74b-4cfb-87e9-36f916d4f590"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:54:56.075127 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.075076 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12a04ce2-d74b-4cfb-87e9-36f916d4f590-kube-api-access-nwsvq" pod "12a04ce2-d74b-4cfb-87e9-36f916d4f590" (UID: "12a04ce2-d74b-4cfb-87e9-36f916d4f590"). InnerVolumeSpecName "kube-api-access-nwsvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:54:56.076089 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.076066 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-metrics-certs" pod "12a04ce2-d74b-4cfb-87e9-36f916d4f590" (UID: "12a04ce2-d74b-4cfb-87e9-36f916d4f590"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:54:56.091630 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.084100 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls" pod "12a04ce2-d74b-4cfb-87e9-36f916d4f590" (UID: "12a04ce2-d74b-4cfb-87e9-36f916d4f590"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:54:56.190744 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.176920 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwsvq\" (UniqueName: \"kubernetes.io/projected/12a04ce2-d74b-4cfb-87e9-36f916d4f590-kube-api-access-nwsvq\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:56.190744 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.176959 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a04ce2-d74b-4cfb-87e9-36f916d4f590-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:56.190744 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.176974 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:56.190744 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.176990 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12a04ce2-d74b-4cfb-87e9-36f916d4f590-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:54:56.236090 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:54:56.236046 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:54:56.310552 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.310460 4399 generic.go:356] "Generic (PLEG): container finished" podID="12a04ce2-d74b-4cfb-87e9-36f916d4f590" containerID="08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058" exitCode=0 Oct 31 15:54:56.310552 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.310525 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" Oct 31 15:54:56.310806 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.310550 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" event={"ID":"12a04ce2-d74b-4cfb-87e9-36f916d4f590","Type":"ContainerDied","Data":"08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058"} Oct 31 15:54:56.310806 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.310600 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg" event={"ID":"12a04ce2-d74b-4cfb-87e9-36f916d4f590","Type":"ContainerDied","Data":"654fded306ab79d2752e2fc1d1af506679ced29df92fa244126d77f1e9bea300"} Oct 31 15:54:56.310806 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.310622 4399 scope.go:122] "RemoveContainer" containerID="08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058" Oct 31 15:54:56.325488 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.325446 4399 scope.go:122] "RemoveContainer" containerID="08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058" Oct 31 15:54:56.341215 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:54:56.337289 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058\": container with ID starting with 08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058 not found: ID does not exist" containerID="08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058" Oct 31 15:54:56.341215 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.337346 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058"} err="failed to get container status \"08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058\": rpc error: code = NotFound desc = could not find container \"08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058\": container with ID starting with 08ae3b33fd030abb2f5291baab47ddc75429c2f898d0e598a4293485bfb50058 not found: ID does not exist" Oct 31 15:54:56.508771 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.508730 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg"] Oct 31 15:54:56.511391 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.511365 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cf95c9-557db449-5ljlg"] Oct 31 15:54:56.897442 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:56.897405 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12a04ce2-d74b-4cfb-87e9-36f916d4f590" path="/var/lib/kubelet/pods/12a04ce2-d74b-4cfb-87e9-36f916d4f590/volumes" Oct 31 15:54:59.670716 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:59.670661 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" podUID="fa0a4375-4680-4d0e-a70d-37151b2eb5f6" containerName="payload-pre-processing" containerID="cri-o://28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612" gracePeriod=25 Oct 31 15:54:59.935656 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:54:59.935597 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:55:00.028131 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.028102 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-kube-api-access-zlhs8\" (UniqueName: \"kubernetes.io/projected/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-kube-api-access-zlhs8\") pod \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\" (UID: \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\") " Oct 31 15:55:00.028296 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.028260 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-plugins-config-volume\") pod \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\" (UID: \"fa0a4375-4680-4d0e-a70d-37151b2eb5f6\") " Oct 31 15:55:00.028674 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.028650 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-plugins-config-volume" pod "fa0a4375-4680-4d0e-a70d-37151b2eb5f6" (UID: "fa0a4375-4680-4d0e-a70d-37151b2eb5f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:55:00.030377 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.030353 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-kube-api-access-zlhs8" pod "fa0a4375-4680-4d0e-a70d-37151b2eb5f6" (UID: "fa0a4375-4680-4d0e-a70d-37151b2eb5f6"). InnerVolumeSpecName "kube-api-access-zlhs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:55:00.129411 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.129381 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:00.129411 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.129408 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhs8\" (UniqueName: \"kubernetes.io/projected/fa0a4375-4680-4d0e-a70d-37151b2eb5f6-kube-api-access-zlhs8\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:00.335570 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.335534 4399 generic.go:356] "Generic (PLEG): container finished" podID="fa0a4375-4680-4d0e-a70d-37151b2eb5f6" containerID="28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612" exitCode=0 Oct 31 15:55:00.335834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.335580 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" Oct 31 15:55:00.335834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.335590 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" event={"ID":"fa0a4375-4680-4d0e-a70d-37151b2eb5f6","Type":"ContainerDied","Data":"28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612"} Oct 31 15:55:00.335834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.335631 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5" event={"ID":"fa0a4375-4680-4d0e-a70d-37151b2eb5f6","Type":"ContainerDied","Data":"b05f93cf0f27a5f86f7884d5d5d83a0104bb8ddcfe12eb22f3758eb6c7628cf4"} Oct 31 15:55:00.335834 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.335653 4399 scope.go:122] "RemoveContainer" containerID="28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612" Oct 31 15:55:00.346190 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.346170 4399 scope.go:122] "RemoveContainer" containerID="28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612" Oct 31 15:55:00.346539 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:55:00.346519 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612\": container with ID starting with 28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612 not found: ID does not exist" containerID="28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612" Oct 31 15:55:00.346586 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.346549 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612"} err="failed to get container status \"28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612\": rpc error: code = NotFound desc = could not find container \"28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612\": container with ID starting with 28afcf37180e32a782d125f75082aba60d93b3905b7d4f4c758315b3cdf78612 not found: ID does not exist" Oct 31 15:55:00.500041 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.499994 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5"] Oct 31 15:55:00.549266 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.549235 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf95c9-5b79f7cc67-md8p5"] Oct 31 15:55:00.897236 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:00.897190 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0a4375-4680-4d0e-a70d-37151b2eb5f6" path="/var/lib/kubelet/pods/fa0a4375-4680-4d0e-a70d-37151b2eb5f6/volumes" Oct 31 15:55:20.567916 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:20.567884 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62"] Oct 31 15:55:20.568604 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:20.568107 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" podUID="f7afc771-470c-4206-b9f2-5b97c17c6737" containerName="maas-api" containerID="cri-o://c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc" gracePeriod=30 Oct 31 15:55:20.878742 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:20.878679 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:55:20.987976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:20.980085 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-maas-api-tls\") pod \"f7afc771-470c-4206-b9f2-5b97c17c6737\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " Oct 31 15:55:20.987976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:20.980247 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7afc771-470c-4206-b9f2-5b97c17c6737-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7afc771-470c-4206-b9f2-5b97c17c6737-openshift-service-ca\") pod \"f7afc771-470c-4206-b9f2-5b97c17c6737\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " Oct 31 15:55:20.987976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:20.980391 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7afc771-470c-4206-b9f2-5b97c17c6737-kube-api-access-7s2tw\" (UniqueName: \"kubernetes.io/projected/f7afc771-470c-4206-b9f2-5b97c17c6737-kube-api-access-7s2tw\") pod \"f7afc771-470c-4206-b9f2-5b97c17c6737\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " Oct 31 15:55:20.987976 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:20.980462 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-metrics-certs\") pod \"f7afc771-470c-4206-b9f2-5b97c17c6737\" (UID: \"f7afc771-470c-4206-b9f2-5b97c17c6737\") " Oct 31 15:55:21.000333 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.000255 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7afc771-470c-4206-b9f2-5b97c17c6737-openshift-service-ca" pod "f7afc771-470c-4206-b9f2-5b97c17c6737" (UID: "f7afc771-470c-4206-b9f2-5b97c17c6737"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:55:21.006956 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.006901 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-maas-api-tls" pod "f7afc771-470c-4206-b9f2-5b97c17c6737" (UID: "f7afc771-470c-4206-b9f2-5b97c17c6737"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:55:21.008914 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.008878 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-metrics-certs" pod "f7afc771-470c-4206-b9f2-5b97c17c6737" (UID: "f7afc771-470c-4206-b9f2-5b97c17c6737"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:55:21.009074 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.008973 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7afc771-470c-4206-b9f2-5b97c17c6737-kube-api-access-7s2tw" pod "f7afc771-470c-4206-b9f2-5b97c17c6737" (UID: "f7afc771-470c-4206-b9f2-5b97c17c6737"). InnerVolumeSpecName "kube-api-access-7s2tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:55:21.082448 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.082413 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:21.082448 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.082443 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7afc771-470c-4206-b9f2-5b97c17c6737-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:21.082448 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.082454 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s2tw\" (UniqueName: \"kubernetes.io/projected/f7afc771-470c-4206-b9f2-5b97c17c6737-kube-api-access-7s2tw\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:21.082820 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.082463 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f7afc771-470c-4206-b9f2-5b97c17c6737-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:21.463365 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.463319 4399 generic.go:356] "Generic (PLEG): container finished" podID="f7afc771-470c-4206-b9f2-5b97c17c6737" containerID="c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc" exitCode=0 Oct 31 15:55:21.463559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.463374 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" Oct 31 15:55:21.463559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.463391 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" event={"ID":"f7afc771-470c-4206-b9f2-5b97c17c6737","Type":"ContainerDied","Data":"c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc"} Oct 31 15:55:21.463559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.463449 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62" event={"ID":"f7afc771-470c-4206-b9f2-5b97c17c6737","Type":"ContainerDied","Data":"4e1cfe21864e360727a79d7e82bc8b87d4a2164de47ec66d2c4f0620ce776a44"} Oct 31 15:55:21.463559 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.463477 4399 scope.go:122] "RemoveContainer" containerID="c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc" Oct 31 15:55:21.524048 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.523210 4399 scope.go:122] "RemoveContainer" containerID="c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc" Oct 31 15:55:21.524400 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:55:21.524313 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc\": container with ID starting with c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc not found: ID does not exist" containerID="c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc" Oct 31 15:55:21.524400 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.524358 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc"} err="failed to get container status \"c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc\": rpc error: code = NotFound desc = could not find container \"c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc\": container with ID starting with c26a72b9cafd3504f202af6e41668932a4343266fd096e5d5b8d2a5f65d534bc not found: ID does not exist" Oct 31 15:55:21.626672 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.626638 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62"] Oct 31 15:55:21.629329 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:21.629307 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c5ee32-746b85497-qsf62"] Oct 31 15:55:22.897103 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:22.897062 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7afc771-470c-4206-b9f2-5b97c17c6737" path="/var/lib/kubelet/pods/f7afc771-470c-4206-b9f2-5b97c17c6737/volumes" Oct 31 15:55:26.215658 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:55:26.215622 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:55:29.625992 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:29.625954 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk"] Oct 31 15:55:29.626495 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:29.626220 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" containerID="cri-o://88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84" gracePeriod=30 Oct 31 15:55:32.880076 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:32.880038 4399 patch_prober.go:29] interesting pod/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:55:32.880473 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:32.880101 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:55:34.909610 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.909586 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:55:34.978081 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978053 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-data\") pod \"f83e1c67-842d-4759-8e56-7d14d56f4e51\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " Oct 31 15:55:34.978212 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978088 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-envoy\") pod \"f83e1c67-842d-4759-8e56-7d14d56f4e51\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " Oct 31 15:55:34.978212 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978133 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-podinfo\") pod \"f83e1c67-842d-4759-8e56-7d14d56f4e51\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " Oct 31 15:55:34.978352 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978235 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-credential-socket\") pod \"f83e1c67-842d-4759-8e56-7d14d56f4e51\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " Oct 31 15:55:34.978352 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978275 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-socket\") pod \"f83e1c67-842d-4759-8e56-7d14d56f4e51\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " Oct 31 15:55:34.978352 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978307 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-token\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-token\") pod \"f83e1c67-842d-4759-8e56-7d14d56f4e51\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " Oct 31 15:55:34.978352 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978311 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-data" pod "f83e1c67-842d-4759-8e56-7d14d56f4e51" (UID: "f83e1c67-842d-4759-8e56-7d14d56f4e51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:55:34.978568 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978352 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-kube-api-access-9wptf\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-kube-api-access-9wptf\") pod \"f83e1c67-842d-4759-8e56-7d14d56f4e51\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " Oct 31 15:55:34.978568 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978385 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83e1c67-842d-4759-8e56-7d14d56f4e51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83e1c67-842d-4759-8e56-7d14d56f4e51-istiod-ca-cert\") pod \"f83e1c67-842d-4759-8e56-7d14d56f4e51\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " Oct 31 15:55:34.978568 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978475 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-certs\") pod \"f83e1c67-842d-4759-8e56-7d14d56f4e51\" (UID: \"f83e1c67-842d-4759-8e56-7d14d56f4e51\") " Oct 31 15:55:34.978568 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978535 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-credential-socket" pod "f83e1c67-842d-4759-8e56-7d14d56f4e51" (UID: "f83e1c67-842d-4759-8e56-7d14d56f4e51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:55:34.978568 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978557 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-socket" pod "f83e1c67-842d-4759-8e56-7d14d56f4e51" (UID: "f83e1c67-842d-4759-8e56-7d14d56f4e51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:55:34.978912 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978850 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-certs" pod "f83e1c67-842d-4759-8e56-7d14d56f4e51" (UID: "f83e1c67-842d-4759-8e56-7d14d56f4e51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:55:34.978912 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.978878 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83e1c67-842d-4759-8e56-7d14d56f4e51-istiod-ca-cert" pod "f83e1c67-842d-4759-8e56-7d14d56f4e51" (UID: "f83e1c67-842d-4759-8e56-7d14d56f4e51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:55:34.979643 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.979619 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:34.979845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.979818 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:34.979845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.979847 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f83e1c67-842d-4759-8e56-7d14d56f4e51-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:34.980033 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.979862 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:34.980033 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.979877 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:34.980679 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.980642 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-envoy" pod "f83e1c67-842d-4759-8e56-7d14d56f4e51" (UID: "f83e1c67-842d-4759-8e56-7d14d56f4e51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:55:34.980863 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.980841 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-kube-api-access-9wptf" pod "f83e1c67-842d-4759-8e56-7d14d56f4e51" (UID: "f83e1c67-842d-4759-8e56-7d14d56f4e51"). InnerVolumeSpecName "kube-api-access-9wptf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:55:34.980934 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.980912 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-token" pod "f83e1c67-842d-4759-8e56-7d14d56f4e51" (UID: "f83e1c67-842d-4759-8e56-7d14d56f4e51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:55:34.981021 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:34.981007 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-podinfo" pod "f83e1c67-842d-4759-8e56-7d14d56f4e51" (UID: "f83e1c67-842d-4759-8e56-7d14d56f4e51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:55:35.080825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.080783 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:35.080825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.080823 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:35.080825 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.080832 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:35.081046 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.080843 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wptf\" (UniqueName: \"kubernetes.io/projected/f83e1c67-842d-4759-8e56-7d14d56f4e51-kube-api-access-9wptf\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:55:35.574862 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.574830 4399 generic.go:356] "Generic (PLEG): container finished" podID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerID="88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84" exitCode=0 Oct 31 15:55:35.575087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.574894 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" Oct 31 15:55:35.575087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.574923 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" event={"ID":"f83e1c67-842d-4759-8e56-7d14d56f4e51","Type":"ContainerDied","Data":"88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84"} Oct 31 15:55:35.575087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.574978 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk" event={"ID":"f83e1c67-842d-4759-8e56-7d14d56f4e51","Type":"ContainerDied","Data":"a540a7486d56b9a3893e87e8877a28ae68a54492eded3813030dbe4abc7002f2"} Oct 31 15:55:35.575087 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.574999 4399 scope.go:122] "RemoveContainer" containerID="88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84" Oct 31 15:55:35.586525 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.586504 4399 scope.go:122] "RemoveContainer" containerID="e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1" Oct 31 15:55:35.595926 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.595913 4399 scope.go:122] "RemoveContainer" containerID="88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84" Oct 31 15:55:35.596212 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:55:35.596187 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84\": container with ID starting with 88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84 not found: ID does not exist" containerID="88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84" Oct 31 15:55:35.596264 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.596220 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84"} err="failed to get container status \"88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84\": rpc error: code = NotFound desc = could not find container \"88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84\": container with ID starting with 88d9e187c4f32661bb4d8ba7b004ea866d5ea794b15a0abebded433076bc2c84 not found: ID does not exist" Oct 31 15:55:35.596264 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.596236 4399 scope.go:122] "RemoveContainer" containerID="e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1" Oct 31 15:55:35.596486 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:55:35.596468 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1\": container with ID starting with e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1 not found: ID does not exist" containerID="e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1" Oct 31 15:55:35.596544 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.596493 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1"} err="failed to get container status \"e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1\": rpc error: code = NotFound desc = could not find container \"e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1\": container with ID starting with e49d706039a1be13815bd6e9f2f0d6f7cb314edfc44667a6bf0b4a013faaedd1 not found: ID does not exist" Oct 31 15:55:35.738323 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.738291 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk"] Oct 31 15:55:35.791094 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:35.791066 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c5ee32-openshift-default-5855644c75-gv4sk"] Oct 31 15:55:36.896998 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:55:36.896966 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" path="/var/lib/kubelet/pods/f83e1c67-842d-4759-8e56-7d14d56f4e51/volumes" Oct 31 15:55:56.207179 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:55:56.207135 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:56:26.226058 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:56:26.226011 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:56:56.207329 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:56:56.207292 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:57:08.455109 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.455029 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8"] Oct 31 15:57:08.455508 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.455280 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" podUID="519232b6-f149-4927-ad3d-7d2224fd47f9" containerName="maas-api" containerID="cri-o://ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799" gracePeriod=30 Oct 31 15:57:08.763910 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.763886 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:57:08.916304 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.916270 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/519232b6-f149-4927-ad3d-7d2224fd47f9-kube-api-access-2wzg5\" (UniqueName: \"kubernetes.io/projected/519232b6-f149-4927-ad3d-7d2224fd47f9-kube-api-access-2wzg5\") pod \"519232b6-f149-4927-ad3d-7d2224fd47f9\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " Oct 31 15:57:08.916493 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.916456 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls\") pod \"519232b6-f149-4927-ad3d-7d2224fd47f9\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " Oct 31 15:57:08.916613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.916594 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-metrics-certs\") pod \"519232b6-f149-4927-ad3d-7d2224fd47f9\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " Oct 31 15:57:08.916828 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.916806 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/519232b6-f149-4927-ad3d-7d2224fd47f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519232b6-f149-4927-ad3d-7d2224fd47f9-openshift-service-ca\") pod \"519232b6-f149-4927-ad3d-7d2224fd47f9\" (UID: \"519232b6-f149-4927-ad3d-7d2224fd47f9\") " Oct 31 15:57:08.917406 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.917285 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/519232b6-f149-4927-ad3d-7d2224fd47f9-openshift-service-ca" pod "519232b6-f149-4927-ad3d-7d2224fd47f9" (UID: "519232b6-f149-4927-ad3d-7d2224fd47f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:57:08.918031 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.918008 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/519232b6-f149-4927-ad3d-7d2224fd47f9-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:08.919035 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.919003 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls" pod "519232b6-f149-4927-ad3d-7d2224fd47f9" (UID: "519232b6-f149-4927-ad3d-7d2224fd47f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:57:08.919877 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.919840 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/519232b6-f149-4927-ad3d-7d2224fd47f9-kube-api-access-2wzg5" pod "519232b6-f149-4927-ad3d-7d2224fd47f9" (UID: "519232b6-f149-4927-ad3d-7d2224fd47f9"). InnerVolumeSpecName "kube-api-access-2wzg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:08.920390 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:08.920371 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-metrics-certs" pod "519232b6-f149-4927-ad3d-7d2224fd47f9" (UID: "519232b6-f149-4927-ad3d-7d2224fd47f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:57:09.020777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.020035 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wzg5\" (UniqueName: \"kubernetes.io/projected/519232b6-f149-4927-ad3d-7d2224fd47f9-kube-api-access-2wzg5\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:09.020777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.020069 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:09.020777 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.020084 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/519232b6-f149-4927-ad3d-7d2224fd47f9-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:09.263770 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.263734 4399 generic.go:356] "Generic (PLEG): container finished" podID="519232b6-f149-4927-ad3d-7d2224fd47f9" containerID="ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799" exitCode=0 Oct 31 15:57:09.263770 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.263771 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" Oct 31 15:57:09.264004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.263817 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" event={"ID":"519232b6-f149-4927-ad3d-7d2224fd47f9","Type":"ContainerDied","Data":"ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799"} Oct 31 15:57:09.264004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.263877 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8" event={"ID":"519232b6-f149-4927-ad3d-7d2224fd47f9","Type":"ContainerDied","Data":"27c8d211efbe00f7a70993933de8fab4377790c32e74e2ea8fa71a220ef1fe2a"} Oct 31 15:57:09.264004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.263901 4399 scope.go:122] "RemoveContainer" containerID="ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799" Oct 31 15:57:09.273791 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.273770 4399 scope.go:122] "RemoveContainer" containerID="ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799" Oct 31 15:57:09.274055 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:57:09.274038 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799\": container with ID starting with ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799 not found: ID does not exist" containerID="ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799" Oct 31 15:57:09.274127 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.274066 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799"} err="failed to get container status \"ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799\": rpc error: code = NotFound desc = could not find container \"ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799\": container with ID starting with ec82cc065fab02ebbf76876fc05e429a8f1f821449fe944f4be34c6d5d44e799 not found: ID does not exist" Oct 31 15:57:09.446327 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.446298 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8"] Oct 31 15:57:09.448932 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:09.448900 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-21111f-675d77f7c-7s7m8"] Oct 31 15:57:10.897273 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:10.897235 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="519232b6-f149-4927-ad3d-7d2224fd47f9" path="/var/lib/kubelet/pods/519232b6-f149-4927-ad3d-7d2224fd47f9/volumes" Oct 31 15:57:26.207277 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:57:26.207241 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:57:33.924419 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:33.924377 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv"] Oct 31 15:57:33.925056 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:33.924609 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" podUID="75a42016-5d8e-493b-934d-fd149a21b752" containerName="maas-api" containerID="cri-o://13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98" gracePeriod=30 Oct 31 15:57:34.217223 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.217192 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:57:34.334267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.334223 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75a42016-5d8e-493b-934d-fd149a21b752-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a42016-5d8e-493b-934d-fd149a21b752-openshift-service-ca\") pod \"75a42016-5d8e-493b-934d-fd149a21b752\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " Oct 31 15:57:34.342257 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.342225 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-metrics-certs\") pod \"75a42016-5d8e-493b-934d-fd149a21b752\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " Oct 31 15:57:34.343287 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.343216 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75a42016-5d8e-493b-934d-fd149a21b752-kube-api-access-thqvn\" (UniqueName: \"kubernetes.io/projected/75a42016-5d8e-493b-934d-fd149a21b752-kube-api-access-thqvn\") pod \"75a42016-5d8e-493b-934d-fd149a21b752\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " Oct 31 15:57:34.345267 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.345233 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls\") pod \"75a42016-5d8e-493b-934d-fd149a21b752\" (UID: \"75a42016-5d8e-493b-934d-fd149a21b752\") " Oct 31 15:57:34.346227 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.338668 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a42016-5d8e-493b-934d-fd149a21b752-openshift-service-ca" pod "75a42016-5d8e-493b-934d-fd149a21b752" (UID: "75a42016-5d8e-493b-934d-fd149a21b752"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:57:34.354050 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.354002 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls" pod "75a42016-5d8e-493b-934d-fd149a21b752" (UID: "75a42016-5d8e-493b-934d-fd149a21b752"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:57:34.357426 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.357373 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-metrics-certs" pod "75a42016-5d8e-493b-934d-fd149a21b752" (UID: "75a42016-5d8e-493b-934d-fd149a21b752"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 15:57:34.364597 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.364436 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a42016-5d8e-493b-934d-fd149a21b752-kube-api-access-thqvn" pod "75a42016-5d8e-493b-934d-fd149a21b752" (UID: "75a42016-5d8e-493b-934d-fd149a21b752"). InnerVolumeSpecName "kube-api-access-thqvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:57:34.412097 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.412066 4399 generic.go:356] "Generic (PLEG): container finished" podID="75a42016-5d8e-493b-934d-fd149a21b752" containerID="13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98" exitCode=0 Oct 31 15:57:34.412214 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.412123 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" Oct 31 15:57:34.412214 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.412153 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" event={"ID":"75a42016-5d8e-493b-934d-fd149a21b752","Type":"ContainerDied","Data":"13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98"} Oct 31 15:57:34.412357 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.412216 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv" event={"ID":"75a42016-5d8e-493b-934d-fd149a21b752","Type":"ContainerDied","Data":"f6c49cdeeacd71ecc1c4c0d666a322fef47158fa773e7d68beb8a897b5dfc974"} Oct 31 15:57:34.412357 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.412247 4399 scope.go:122] "RemoveContainer" containerID="13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98" Oct 31 15:57:34.425005 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.424985 4399 scope.go:122] "RemoveContainer" containerID="13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98" Oct 31 15:57:34.425490 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:57:34.425471 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98\": container with ID starting with 13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98 not found: ID does not exist" containerID="13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98" Oct 31 15:57:34.425568 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.425499 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98"} err="failed to get container status \"13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98\": rpc error: code = NotFound desc = could not find container \"13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98\": container with ID starting with 13ed4eab4878d1e3aacf651238fe97d5b9954295ed4dfa284f7e450e3e435a98 not found: ID does not exist" Oct 31 15:57:34.455721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.455651 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:34.455721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.455690 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75a42016-5d8e-493b-934d-fd149a21b752-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:34.455721 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.455728 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75a42016-5d8e-493b-934d-fd149a21b752-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:34.456055 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.455741 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thqvn\" (UniqueName: \"kubernetes.io/projected/75a42016-5d8e-493b-934d-fd149a21b752-kube-api-access-thqvn\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:57:34.577789 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.577754 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv"] Oct 31 15:57:34.580251 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.580231 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-46bf36-74645fb55-l6fcv"] Oct 31 15:57:34.911907 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:34.911832 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75a42016-5d8e-493b-934d-fd149a21b752" path="/var/lib/kubelet/pods/75a42016-5d8e-493b-934d-fd149a21b752/volumes" Oct 31 15:57:54.175908 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:54.175877 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz"] Oct 31 15:57:54.279546 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:54.279510 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8"] Oct 31 15:57:56.209237 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:57:56.209200 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:57:59.176214 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:59.176176 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" podUID="aad61f6f-8703-413b-8d9e-7538c8804de8" containerName="payload-processing" containerID="cri-o://8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb" gracePeriod=25 Oct 31 15:57:59.280778 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:57:59.280720 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" podUID="87b7090c-0029-4f6c-ac6c-bd4810338241" containerName="payload-pre-processing" containerID="cri-o://1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1" gracePeriod=25 Oct 31 15:58:07.216842 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:07.216800 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" podUID="87b7090c-0029-4f6c-ac6c-bd4810338241" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:58:08.225837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:08.225795 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" podUID="aad61f6f-8703-413b-8d9e-7538c8804de8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:58:17.215942 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:17.215908 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" podUID="87b7090c-0029-4f6c-ac6c-bd4810338241" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:58:18.220484 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:18.220448 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" podUID="aad61f6f-8703-413b-8d9e-7538c8804de8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 15:58:19.473733 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.473709 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 15:58:19.537120 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.537097 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 15:58:19.566437 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.566410 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aad61f6f-8703-413b-8d9e-7538c8804de8-kube-api-access-2jcvf\" (UniqueName: \"kubernetes.io/projected/aad61f6f-8703-413b-8d9e-7538c8804de8-kube-api-access-2jcvf\") pod \"aad61f6f-8703-413b-8d9e-7538c8804de8\" (UID: \"aad61f6f-8703-413b-8d9e-7538c8804de8\") " Oct 31 15:58:19.566691 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.566672 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aad61f6f-8703-413b-8d9e-7538c8804de8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad61f6f-8703-413b-8d9e-7538c8804de8-plugins-config-volume\") pod \"aad61f6f-8703-413b-8d9e-7538c8804de8\" (UID: \"aad61f6f-8703-413b-8d9e-7538c8804de8\") " Oct 31 15:58:19.567007 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.566984 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aad61f6f-8703-413b-8d9e-7538c8804de8-plugins-config-volume" pod "aad61f6f-8703-413b-8d9e-7538c8804de8" (UID: "aad61f6f-8703-413b-8d9e-7538c8804de8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:58:19.567163 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.567146 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aad61f6f-8703-413b-8d9e-7538c8804de8-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:19.568496 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.568464 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aad61f6f-8703-413b-8d9e-7538c8804de8-kube-api-access-2jcvf" pod "aad61f6f-8703-413b-8d9e-7538c8804de8" (UID: "aad61f6f-8703-413b-8d9e-7538c8804de8"). InnerVolumeSpecName "kube-api-access-2jcvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:58:19.668753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.668715 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87b7090c-0029-4f6c-ac6c-bd4810338241-kube-api-access-m55f8\" (UniqueName: \"kubernetes.io/projected/87b7090c-0029-4f6c-ac6c-bd4810338241-kube-api-access-m55f8\") pod \"87b7090c-0029-4f6c-ac6c-bd4810338241\" (UID: \"87b7090c-0029-4f6c-ac6c-bd4810338241\") " Oct 31 15:58:19.668913 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.668786 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87b7090c-0029-4f6c-ac6c-bd4810338241-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b7090c-0029-4f6c-ac6c-bd4810338241-plugins-config-volume\") pod \"87b7090c-0029-4f6c-ac6c-bd4810338241\" (UID: \"87b7090c-0029-4f6c-ac6c-bd4810338241\") " Oct 31 15:58:19.669174 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.669144 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87b7090c-0029-4f6c-ac6c-bd4810338241-plugins-config-volume" pod "87b7090c-0029-4f6c-ac6c-bd4810338241" (UID: "87b7090c-0029-4f6c-ac6c-bd4810338241"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:58:19.669326 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.669264 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87b7090c-0029-4f6c-ac6c-bd4810338241-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:19.669326 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.669284 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jcvf\" (UniqueName: \"kubernetes.io/projected/aad61f6f-8703-413b-8d9e-7538c8804de8-kube-api-access-2jcvf\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:19.670866 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.670844 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87b7090c-0029-4f6c-ac6c-bd4810338241-kube-api-access-m55f8" pod "87b7090c-0029-4f6c-ac6c-bd4810338241" (UID: "87b7090c-0029-4f6c-ac6c-bd4810338241"). InnerVolumeSpecName "kube-api-access-m55f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:58:19.680186 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.680159 4399 generic.go:356] "Generic (PLEG): container finished" podID="87b7090c-0029-4f6c-ac6c-bd4810338241" containerID="1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1" exitCode=0 Oct 31 15:58:19.680314 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.680201 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" Oct 31 15:58:19.680314 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.680251 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" event={"ID":"87b7090c-0029-4f6c-ac6c-bd4810338241","Type":"ContainerDied","Data":"1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1"} Oct 31 15:58:19.680459 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.680311 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8" event={"ID":"87b7090c-0029-4f6c-ac6c-bd4810338241","Type":"ContainerDied","Data":"36dc044304828ca96aba0fd580276fb420da7ce057f96f78a2d24fde7974b014"} Oct 31 15:58:19.680459 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.680330 4399 scope.go:122] "RemoveContainer" containerID="1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1" Oct 31 15:58:19.681786 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.681761 4399 generic.go:356] "Generic (PLEG): container finished" podID="aad61f6f-8703-413b-8d9e-7538c8804de8" containerID="8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb" exitCode=0 Oct 31 15:58:19.681897 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.681816 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" Oct 31 15:58:19.681897 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.681850 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" event={"ID":"aad61f6f-8703-413b-8d9e-7538c8804de8","Type":"ContainerDied","Data":"8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb"} Oct 31 15:58:19.681897 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.681896 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz" event={"ID":"aad61f6f-8703-413b-8d9e-7538c8804de8","Type":"ContainerDied","Data":"a35f4ea96134fde9cf146e7184ae5718a71a044eda46f9863ca42e79fe234b7c"} Oct 31 15:58:19.690045 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.690027 4399 scope.go:122] "RemoveContainer" containerID="1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1" Oct 31 15:58:19.690404 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:58:19.690382 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1\": container with ID starting with 1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1 not found: ID does not exist" containerID="1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1" Oct 31 15:58:19.690512 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.690411 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1"} err="failed to get container status \"1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1\": rpc error: code = NotFound desc = could not find container \"1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1\": container with ID starting with 1ab3257d534b2b175e78d9dc1dcc4eb9c6ee49828d4026c1440ef3f8b83344c1 not found: ID does not exist" Oct 31 15:58:19.690512 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.690428 4399 scope.go:122] "RemoveContainer" containerID="8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb" Oct 31 15:58:19.699229 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.699211 4399 scope.go:122] "RemoveContainer" containerID="8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb" Oct 31 15:58:19.699483 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:58:19.699467 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb\": container with ID starting with 8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb not found: ID does not exist" containerID="8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb" Oct 31 15:58:19.699527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.699489 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb"} err="failed to get container status \"8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb\": rpc error: code = NotFound desc = could not find container \"8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb\": container with ID starting with 8f98516ba0145be97c3d8a9fcb45e3b233e1cdd8b4fbd9e11b9ec8787ddf26eb not found: ID does not exist" Oct 31 15:58:19.770340 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.770310 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m55f8\" (UniqueName: \"kubernetes.io/projected/87b7090c-0029-4f6c-ac6c-bd4810338241-kube-api-access-m55f8\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:58:19.820565 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.820535 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz"] Oct 31 15:58:19.823173 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.823152 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-57815b-79cb7f9f47-l4xwz"] Oct 31 15:58:19.928921 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.928881 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8"] Oct 31 15:58:19.931333 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:19.931310 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-57815b-cc5645659-svjp8"] Oct 31 15:58:20.896004 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:20.895974 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87b7090c-0029-4f6c-ac6c-bd4810338241" path="/var/lib/kubelet/pods/87b7090c-0029-4f6c-ac6c-bd4810338241/volumes" Oct 31 15:58:20.896379 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:20.896330 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aad61f6f-8703-413b-8d9e-7538c8804de8" path="/var/lib/kubelet/pods/aad61f6f-8703-413b-8d9e-7538c8804de8/volumes" Oct 31 15:58:26.206264 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:58:26.206225 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:58:56.206879 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:58:56.206843 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:58:58.571381 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:58.571290 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg"] Oct 31 15:58:58.571824 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:58:58.571497 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" containerID="cri-o://74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37" gracePeriod=30 Oct 31 15:59:01.112987 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:01.112954 4399 patch_prober.go:29] interesting pod/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 15:59:01.113366 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:01.113027 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 15:59:03.832626 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:03.832602 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:59:04.007613 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.007580 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istiod-ca-cert\") pod \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " Oct 31 15:59:04.007837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.007643 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-data\") pod \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " Oct 31 15:59:04.007837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.007741 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-envoy\") pod \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " Oct 31 15:59:04.007837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.007788 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-podinfo\") pod \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " Oct 31 15:59:04.007837 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.007817 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-certs\") pod \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " Oct 31 15:59:04.008084 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.007858 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-socket\") pod \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " Oct 31 15:59:04.008084 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.007890 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-credential-socket\") pod \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " Oct 31 15:59:04.008084 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.007912 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-token\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-token\") pod \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " Oct 31 15:59:04.008084 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.007974 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-kube-api-access-l8l9q\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-kube-api-access-l8l9q\") pod \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\" (UID: \"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37\") " Oct 31 15:59:04.009391 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.009262 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-certs" pod "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" (UID: "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.010133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.009584 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-socket" pod "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" (UID: "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.010133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.009766 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-credential-socket" pod "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" (UID: "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.010133 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.009802 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-data" pod "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" (UID: "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.010353 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.010327 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istiod-ca-cert" pod "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" (UID: "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 15:59:04.011259 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.011229 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-kube-api-access-l8l9q" pod "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" (UID: "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37"). InnerVolumeSpecName "kube-api-access-l8l9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:59:04.012753 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.012717 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-envoy" pod "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" (UID: "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 15:59:04.012912 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.012884 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-podinfo" pod "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" (UID: "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 15:59:04.013398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.013225 4399 generic.go:356] "Generic (PLEG): container finished" podID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerID="74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37" exitCode=0 Oct 31 15:59:04.013398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.013337 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" Oct 31 15:59:04.013398 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.013354 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" event={"ID":"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37","Type":"ContainerDied","Data":"74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37"} Oct 31 15:59:04.013610 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.013336 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-token" pod "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" (UID: "6e96efc2-aa5d-49b1-87b4-6fb9b01acf37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 15:59:04.013610 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.013425 4399 scope.go:122] "RemoveContainer" containerID="74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37" Oct 31 15:59:04.013610 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.013413 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg" event={"ID":"6e96efc2-aa5d-49b1-87b4-6fb9b01acf37","Type":"ContainerDied","Data":"b64b77615ea80117c5af78e568bfd6876b5cb5da3193fd40e37d18de123da2a3"} Oct 31 15:59:04.038536 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.038516 4399 scope.go:122] "RemoveContainer" containerID="74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37" Oct 31 15:59:04.038913 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:59:04.038866 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37\": container with ID starting with 74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37 not found: ID does not exist" containerID="74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37" Oct 31 15:59:04.039012 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.038920 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37"} err="failed to get container status \"74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37\": rpc error: code = NotFound desc = could not find container \"74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37\": container with ID starting with 74a9669f657ea5937ffda1b1d8ba5c9e667fdabfc5a7f88b9c7140f741ff5f37 not found: ID does not exist" Oct 31 15:59:04.109500 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.109473 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.109500 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.109499 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.109727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.109508 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.109727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.109517 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.109727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.109526 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.109727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.109534 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.109727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.109542 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.109727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.109550 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.109727 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.109558 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8l9q\" (UniqueName: \"kubernetes.io/projected/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37-kube-api-access-l8l9q\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 15:59:04.178527 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.178490 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg"] Oct 31 15:59:04.180960 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.180939 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-57815b-openshift-default-76c4dbcb57-4l5wg"] Oct 31 15:59:04.895845 ip-10-0-137-160 kubenswrapper[4399]: I1031 15:59:04.895817 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" path="/var/lib/kubelet/pods/6e96efc2-aa5d-49b1-87b4-6fb9b01acf37/volumes" Oct 31 15:59:26.207476 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:59:26.207436 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 15:59:56.206473 ip-10-0-137-160 kubenswrapper[4399]: E1031 15:59:56.206437 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:00:00.213815 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.213784 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt"] Oct 31 16:00:00.214305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214292 4399 cpu_manager.go:414] "containerMap: removing container" podUID="aad61f6f-8703-413b-8d9e-7538c8804de8" containerName="payload-processing" Oct 31 16:00:00.214351 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214306 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aad61f6f-8703-413b-8d9e-7538c8804de8" containerName="payload-processing" Oct 31 16:00:00.214351 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214320 4399 cpu_manager.go:414] "containerMap: removing container" podUID="fd35a41e-27bd-4139-99fa-60f81f538369" containerName="maas-api" Oct 31 16:00:00.214351 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214326 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd35a41e-27bd-4139-99fa-60f81f538369" containerName="maas-api" Oct 31 16:00:00.214351 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214335 4399 cpu_manager.go:414] "containerMap: removing container" podUID="f7afc771-470c-4206-b9f2-5b97c17c6737" containerName="maas-api" Oct 31 16:00:00.214351 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214340 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7afc771-470c-4206-b9f2-5b97c17c6737" containerName="maas-api" Oct 31 16:00:00.214351 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214347 4399 cpu_manager.go:414] "containerMap: removing container" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" Oct 31 16:00:00.214351 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214352 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214359 4399 cpu_manager.go:414] "containerMap: removing container" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214363 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214371 4399 cpu_manager.go:414] "containerMap: removing container" podUID="519232b6-f149-4927-ad3d-7d2224fd47f9" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214375 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="519232b6-f149-4927-ad3d-7d2224fd47f9" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214380 4399 cpu_manager.go:414] "containerMap: removing container" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214384 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214394 4399 cpu_manager.go:414] "containerMap: removing container" podUID="87b7090c-0029-4f6c-ac6c-bd4810338241" containerName="payload-pre-processing" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214399 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87b7090c-0029-4f6c-ac6c-bd4810338241" containerName="payload-pre-processing" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214412 4399 cpu_manager.go:414] "containerMap: removing container" podUID="fa0a4375-4680-4d0e-a70d-37151b2eb5f6" containerName="payload-pre-processing" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214419 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0a4375-4680-4d0e-a70d-37151b2eb5f6" containerName="payload-pre-processing" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214425 4399 cpu_manager.go:414] "containerMap: removing container" podUID="12a04ce2-d74b-4cfb-87e9-36f916d4f590" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214429 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12a04ce2-d74b-4cfb-87e9-36f916d4f590" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214438 4399 cpu_manager.go:414] "containerMap: removing container" podUID="75a42016-5d8e-493b-934d-fd149a21b752" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214442 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75a42016-5d8e-493b-934d-fd149a21b752" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214542 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0a4375-4680-4d0e-a70d-37151b2eb5f6" containerName="payload-pre-processing" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214552 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="12a04ce2-d74b-4cfb-87e9-36f916d4f590" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214559 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75a42016-5d8e-493b-934d-fd149a21b752" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214567 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214574 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214580 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="aad61f6f-8703-413b-8d9e-7538c8804de8" containerName="payload-processing" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214586 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd35a41e-27bd-4139-99fa-60f81f538369" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214592 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7afc771-470c-4206-b9f2-5b97c17c6737" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214598 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e96efc2-aa5d-49b1-87b4-6fb9b01acf37" containerName="istio-proxy" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214605 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="519232b6-f149-4927-ad3d-7d2224fd47f9" containerName="maas-api" Oct 31 16:00:00.214598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214609 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" Oct 31 16:00:00.215372 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.214617 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="87b7090c-0029-4f6c-ac6c-bd4810338241" containerName="payload-pre-processing" Oct 31 16:00:00.219178 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.219163 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:00.315255 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.315170 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtw84\" (UniqueName: \"kubernetes.io/projected/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-kube-api-access-rtw84\") pod \"maas-api-key-cleanup-29891040-d8nwt\" (UID: \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:00.315789 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.315605 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891040-d8nwt\" (UID: \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:00.321780 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.321755 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt"] Oct 31 16:00:00.370828 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.370792 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nzmdf\"" Oct 31 16:00:00.416889 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.416847 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtw84\" (UniqueName: \"kubernetes.io/projected/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-kube-api-access-rtw84\") pod \"maas-api-key-cleanup-29891040-d8nwt\" (UID: \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:00.417124 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.416980 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891040-d8nwt\" (UID: \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:00.417673 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.417652 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29891040-d8nwt\" (UID: \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:00.472974 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.472895 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtw84\" (UniqueName: \"kubernetes.io/projected/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-kube-api-access-rtw84\") pod \"maas-api-key-cleanup-29891040-d8nwt\" (UID: \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:00.532112 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.532069 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:00.911415 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:00.911389 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt"] Oct 31 16:00:00.913883 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:00:00.913856 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1656a55a_6a22_41ad_b2fa_e550b0d94b3a.slice/crio-8003b2849fec95347d8e251056a82b1c7d6a6e428bb46a26095caa72711a5c16 WatchSource:0}: Error finding container 8003b2849fec95347d8e251056a82b1c7d6a6e428bb46a26095caa72711a5c16: Status 404 returned error can't find the container with id 8003b2849fec95347d8e251056a82b1c7d6a6e428bb46a26095caa72711a5c16 Oct 31 16:00:01.338165 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:01.338126 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" event={"ID":"1656a55a-6a22-41ad-b2fa-e550b0d94b3a","Type":"ContainerStarted","Data":"506e45aa906fa25cd36a4c67a97646fa3e656980dde3b8dd0a28a982bfb9374b"} Oct 31 16:00:01.338546 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:01.338189 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" event={"ID":"1656a55a-6a22-41ad-b2fa-e550b0d94b3a","Type":"ContainerStarted","Data":"8003b2849fec95347d8e251056a82b1c7d6a6e428bb46a26095caa72711a5c16"} Oct 31 16:00:01.449833 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:01.449780 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" podStartSLOduration=1.449766519 podStartE2EDuration="1.449766519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:00:01.449086843 +0000 UTC m=+4444.660328256" watchObservedRunningTime="2026-10-31 16:00:01.449766519 +0000 UTC m=+4444.661007965" Oct 31 16:00:02.344949 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:02.344910 4399 generic.go:356] "Generic (PLEG): container finished" podID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerID="506e45aa906fa25cd36a4c67a97646fa3e656980dde3b8dd0a28a982bfb9374b" exitCode=60 Oct 31 16:00:02.345412 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:02.344957 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" event={"ID":"1656a55a-6a22-41ad-b2fa-e550b0d94b3a","Type":"ContainerDied","Data":"506e45aa906fa25cd36a4c67a97646fa3e656980dde3b8dd0a28a982bfb9374b"} Oct 31 16:00:02.345412 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:02.345235 4399 scope.go:122] "RemoveContainer" containerID="506e45aa906fa25cd36a4c67a97646fa3e656980dde3b8dd0a28a982bfb9374b" Oct 31 16:00:03.352191 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:03.352156 4399 generic.go:356] "Generic (PLEG): container finished" podID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerID="ae4e93d9bb1b1c7f442d0712aac5b7041bb2c0f4e2168c74e6909f1b51b6b682" exitCode=60 Oct 31 16:00:03.352622 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:03.352243 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" event={"ID":"1656a55a-6a22-41ad-b2fa-e550b0d94b3a","Type":"ContainerDied","Data":"ae4e93d9bb1b1c7f442d0712aac5b7041bb2c0f4e2168c74e6909f1b51b6b682"} Oct 31 16:00:03.352622 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:03.352305 4399 scope.go:122] "RemoveContainer" containerID="506e45aa906fa25cd36a4c67a97646fa3e656980dde3b8dd0a28a982bfb9374b" Oct 31 16:00:03.352622 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:03.352565 4399 scope.go:122] "RemoveContainer" containerID="ae4e93d9bb1b1c7f442d0712aac5b7041bb2c0f4e2168c74e6909f1b51b6b682" Oct 31 16:00:03.352879 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:00:03.352856 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891040-d8nwt_odh-ai-gateway-infra(1656a55a-6a22-41ad-b2fa-e550b0d94b3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" Oct 31 16:00:04.359622 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:04.359591 4399 scope.go:122] "RemoveContainer" containerID="ae4e93d9bb1b1c7f442d0712aac5b7041bb2c0f4e2168c74e6909f1b51b6b682" Oct 31 16:00:04.360051 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:00:04.359839 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29891040-d8nwt_odh-ai-gateway-infra(1656a55a-6a22-41ad-b2fa-e550b0d94b3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" Oct 31 16:00:15.890390 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:15.890312 4399 scope.go:122] "RemoveContainer" containerID="ae4e93d9bb1b1c7f442d0712aac5b7041bb2c0f4e2168c74e6909f1b51b6b682" Oct 31 16:00:16.437009 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:16.436917 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" event={"ID":"1656a55a-6a22-41ad-b2fa-e550b0d94b3a","Type":"ContainerStarted","Data":"cbd78f69e97e5098e80a920053bb33c6c1e3d1cce9fbe1bc075097066ecdbb83"} Oct 31 16:00:17.448046 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:17.448011 4399 generic.go:356] "Generic (PLEG): container finished" podID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerID="cbd78f69e97e5098e80a920053bb33c6c1e3d1cce9fbe1bc075097066ecdbb83" exitCode=60 Oct 31 16:00:17.448472 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:17.448067 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" event={"ID":"1656a55a-6a22-41ad-b2fa-e550b0d94b3a","Type":"ContainerDied","Data":"cbd78f69e97e5098e80a920053bb33c6c1e3d1cce9fbe1bc075097066ecdbb83"} Oct 31 16:00:17.448472 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:17.448147 4399 scope.go:122] "RemoveContainer" containerID="ae4e93d9bb1b1c7f442d0712aac5b7041bb2c0f4e2168c74e6909f1b51b6b682" Oct 31 16:00:17.448622 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:17.448559 4399 scope.go:122] "RemoveContainer" containerID="cbd78f69e97e5098e80a920053bb33c6c1e3d1cce9fbe1bc075097066ecdbb83" Oct 31 16:00:17.448939 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:00:17.448911 4399 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29891040-d8nwt_odh-ai-gateway-infra(1656a55a-6a22-41ad-b2fa-e550b0d94b3a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" Oct 31 16:00:17.558087 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:17.558061 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt"] Oct 31 16:00:18.613531 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:18.613503 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:18.757265 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:18.757183 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-maas-api-service-ca\") pod \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\" (UID: \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\") " Oct 31 16:00:18.757435 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:18.757357 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-kube-api-access-rtw84\" (UniqueName: \"kubernetes.io/projected/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-kube-api-access-rtw84\") pod \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\" (UID: \"1656a55a-6a22-41ad-b2fa-e550b0d94b3a\") " Oct 31 16:00:18.757592 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:18.757569 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-maas-api-service-ca" pod "1656a55a-6a22-41ad-b2fa-e550b0d94b3a" (UID: "1656a55a-6a22-41ad-b2fa-e550b0d94b3a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:18.757823 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:18.757806 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-maas-api-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:18.759549 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:18.759528 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-kube-api-access-rtw84" pod "1656a55a-6a22-41ad-b2fa-e550b0d94b3a" (UID: "1656a55a-6a22-41ad-b2fa-e550b0d94b3a"). InnerVolumeSpecName "kube-api-access-rtw84". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:18.858371 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:18.858345 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtw84\" (UniqueName: \"kubernetes.io/projected/1656a55a-6a22-41ad-b2fa-e550b0d94b3a-kube-api-access-rtw84\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:19.463720 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:19.463661 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" Oct 31 16:00:19.463720 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:19.463681 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt" event={"ID":"1656a55a-6a22-41ad-b2fa-e550b0d94b3a","Type":"ContainerDied","Data":"8003b2849fec95347d8e251056a82b1c7d6a6e428bb46a26095caa72711a5c16"} Oct 31 16:00:19.463973 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:19.463770 4399 scope.go:122] "RemoveContainer" containerID="cbd78f69e97e5098e80a920053bb33c6c1e3d1cce9fbe1bc075097066ecdbb83" Oct 31 16:00:19.626503 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:19.626472 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt"] Oct 31 16:00:19.677310 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:19.677276 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29891040-d8nwt"] Oct 31 16:00:20.896575 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:20.896546 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" path="/var/lib/kubelet/pods/1656a55a-6a22-41ad-b2fa-e550b0d94b3a/volumes" Oct 31 16:00:26.207533 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:00:26.207489 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:00:32.935013 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:32.934983 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh"] Oct 31 16:00:32.935576 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:32.935213 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" containerID="cri-o://4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab" gracePeriod=30 Oct 31 16:00:33.648663 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:33.648623 4399 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:00:33.648854 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:33.648693 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:00:38.211519 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.211464 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 16:00:38.212165 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.212145 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-podinfo\") pod \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " Oct 31 16:00:38.212222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.212211 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-data\") pod \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " Oct 31 16:00:38.212546 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.212526 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-data" pod "40ab2c02-b50b-44c0-8073-44d25c2b40aa" (UID: "40ab2c02-b50b-44c0-8073-44d25c2b40aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:38.212602 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.212591 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:38.214172 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.214152 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-podinfo" pod "40ab2c02-b50b-44c0-8073-44d25c2b40aa" (UID: "40ab2c02-b50b-44c0-8073-44d25c2b40aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:00:38.313058 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313030 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-certs\") pod \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " Oct 31 16:00:38.313252 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313084 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-socket\") pod \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " Oct 31 16:00:38.313252 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313103 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-envoy\") pod \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " Oct 31 16:00:38.313252 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313190 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-token\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-token\") pod \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " Oct 31 16:00:38.313252 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313215 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-credential-socket\") pod \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " Oct 31 16:00:38.313486 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313289 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-kube-api-access-6gq89\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-kube-api-access-6gq89\") pod \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " Oct 31 16:00:38.313486 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313315 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istiod-ca-cert\") pod \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\" (UID: \"40ab2c02-b50b-44c0-8073-44d25c2b40aa\") " Oct 31 16:00:38.313486 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313346 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-certs" pod "40ab2c02-b50b-44c0-8073-44d25c2b40aa" (UID: "40ab2c02-b50b-44c0-8073-44d25c2b40aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:38.313486 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313358 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-socket" pod "40ab2c02-b50b-44c0-8073-44d25c2b40aa" (UID: "40ab2c02-b50b-44c0-8073-44d25c2b40aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:38.313736 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313552 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-credential-socket" pod "40ab2c02-b50b-44c0-8073-44d25c2b40aa" (UID: "40ab2c02-b50b-44c0-8073-44d25c2b40aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:38.313736 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313678 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:38.313736 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313720 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:38.313736 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313734 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:38.313896 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313748 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:38.313896 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.313766 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istiod-ca-cert" pod "40ab2c02-b50b-44c0-8073-44d25c2b40aa" (UID: "40ab2c02-b50b-44c0-8073-44d25c2b40aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:00:38.315373 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.315352 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-envoy" pod "40ab2c02-b50b-44c0-8073-44d25c2b40aa" (UID: "40ab2c02-b50b-44c0-8073-44d25c2b40aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:00:38.316080 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.316051 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-token" pod "40ab2c02-b50b-44c0-8073-44d25c2b40aa" (UID: "40ab2c02-b50b-44c0-8073-44d25c2b40aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:38.316172 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.316079 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-kube-api-access-6gq89" pod "40ab2c02-b50b-44c0-8073-44d25c2b40aa" (UID: "40ab2c02-b50b-44c0-8073-44d25c2b40aa"). InnerVolumeSpecName "kube-api-access-6gq89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:00:38.414992 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.414956 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:38.414992 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.414995 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:38.415172 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.415012 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gq89\" (UniqueName: \"kubernetes.io/projected/40ab2c02-b50b-44c0-8073-44d25c2b40aa-kube-api-access-6gq89\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:38.415172 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.415027 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40ab2c02-b50b-44c0-8073-44d25c2b40aa-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:00:38.578621 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.578531 4399 generic.go:356] "Generic (PLEG): container finished" podID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerID="4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab" exitCode=0 Oct 31 16:00:38.578621 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.578581 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" Oct 31 16:00:38.578869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.578610 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" event={"ID":"40ab2c02-b50b-44c0-8073-44d25c2b40aa","Type":"ContainerDied","Data":"4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab"} Oct 31 16:00:38.578869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.578667 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh" event={"ID":"40ab2c02-b50b-44c0-8073-44d25c2b40aa","Type":"ContainerDied","Data":"74b39375c13aec41a99ce9a359be7ac49b81838981120259d67e7aa95ae4b996"} Oct 31 16:00:38.578869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.578725 4399 scope.go:122] "RemoveContainer" containerID="4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab" Oct 31 16:00:38.589272 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.589254 4399 scope.go:122] "RemoveContainer" containerID="044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e" Oct 31 16:00:38.597919 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.597903 4399 scope.go:122] "RemoveContainer" containerID="4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab" Oct 31 16:00:38.598181 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:00:38.598164 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab\": container with ID starting with 4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab not found: ID does not exist" containerID="4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab" Oct 31 16:00:38.598264 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.598189 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab"} err="failed to get container status \"4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab\": rpc error: code = NotFound desc = could not find container \"4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab\": container with ID starting with 4d904b20ee4ecce6a59b8c9178c5f91bfff5a23df6329be965987e60b03e72ab not found: ID does not exist" Oct 31 16:00:38.598264 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.598207 4399 scope.go:122] "RemoveContainer" containerID="044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e" Oct 31 16:00:38.598439 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:00:38.598419 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e\": container with ID starting with 044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e not found: ID does not exist" containerID="044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e" Oct 31 16:00:38.598506 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.598443 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e"} err="failed to get container status \"044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e\": rpc error: code = NotFound desc = could not find container \"044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e\": container with ID starting with 044eecc23a2591133f7af85f8c28a2863214e3c8c50933d18bd901799364616e not found: ID does not exist" Oct 31 16:00:38.749684 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.749638 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh"] Oct 31 16:00:38.749906 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.749723 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c947d9-openshift-default-5ffd8f4877-7jlsh"] Oct 31 16:00:38.896566 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:38.896491 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" path="/var/lib/kubelet/pods/40ab2c02-b50b-44c0-8073-44d25c2b40aa/volumes" Oct 31 16:00:50.920018 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:50.919983 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5"] Oct 31 16:00:55.920677 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:00:55.920636 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" podUID="c486ec0f-d3a7-4625-8bae-1dca830110d2" containerName="payload-processing" containerID="cri-o://1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8" gracePeriod=25 Oct 31 16:00:56.208042 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:00:56.207980 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:01:02.999914 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:02.999862 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" podUID="c486ec0f-d3a7-4625-8bae-1dca830110d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:01:13.000183 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:13.000140 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" podUID="c486ec0f-d3a7-4625-8bae-1dca830110d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:01:16.188893 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.188835 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 16:01:16.268361 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.268336 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c486ec0f-d3a7-4625-8bae-1dca830110d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486ec0f-d3a7-4625-8bae-1dca830110d2-plugins-config-volume\") pod \"c486ec0f-d3a7-4625-8bae-1dca830110d2\" (UID: \"c486ec0f-d3a7-4625-8bae-1dca830110d2\") " Oct 31 16:01:16.268509 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.268389 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c486ec0f-d3a7-4625-8bae-1dca830110d2-kube-api-access-p6k2q\" (UniqueName: \"kubernetes.io/projected/c486ec0f-d3a7-4625-8bae-1dca830110d2-kube-api-access-p6k2q\") pod \"c486ec0f-d3a7-4625-8bae-1dca830110d2\" (UID: \"c486ec0f-d3a7-4625-8bae-1dca830110d2\") " Oct 31 16:01:16.268732 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.268684 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c486ec0f-d3a7-4625-8bae-1dca830110d2-plugins-config-volume" pod "c486ec0f-d3a7-4625-8bae-1dca830110d2" (UID: "c486ec0f-d3a7-4625-8bae-1dca830110d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:01:16.270675 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.270654 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c486ec0f-d3a7-4625-8bae-1dca830110d2-kube-api-access-p6k2q" pod "c486ec0f-d3a7-4625-8bae-1dca830110d2" (UID: "c486ec0f-d3a7-4625-8bae-1dca830110d2"). InnerVolumeSpecName "kube-api-access-p6k2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:01:16.370339 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.370309 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c486ec0f-d3a7-4625-8bae-1dca830110d2-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:01:16.370339 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.370337 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6k2q\" (UniqueName: \"kubernetes.io/projected/c486ec0f-d3a7-4625-8bae-1dca830110d2-kube-api-access-p6k2q\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:01:16.792971 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.792935 4399 generic.go:356] "Generic (PLEG): container finished" podID="c486ec0f-d3a7-4625-8bae-1dca830110d2" containerID="1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8" exitCode=0 Oct 31 16:01:16.792971 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.792978 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" Oct 31 16:01:16.793194 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.793022 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" event={"ID":"c486ec0f-d3a7-4625-8bae-1dca830110d2","Type":"ContainerDied","Data":"1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8"} Oct 31 16:01:16.793194 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.793074 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5" event={"ID":"c486ec0f-d3a7-4625-8bae-1dca830110d2","Type":"ContainerDied","Data":"8f9bc7e62218953e193ac1d6561ec991859c8bd69d2afdacb03e576438835f19"} Oct 31 16:01:16.793194 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.793101 4399 scope.go:122] "RemoveContainer" containerID="1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8" Oct 31 16:01:16.803353 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.803336 4399 scope.go:122] "RemoveContainer" containerID="1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8" Oct 31 16:01:16.803636 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:01:16.803611 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8\": container with ID starting with 1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8 not found: ID does not exist" containerID="1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8" Oct 31 16:01:16.803750 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.803641 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8"} err="failed to get container status \"1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8\": rpc error: code = NotFound desc = could not find container \"1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8\": container with ID starting with 1e16138c8267cc6dcc88bdda9ca557a96c39778fd703c12abd5baf0eb4d9f8c8 not found: ID does not exist" Oct 31 16:01:16.956226 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:16.956193 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5"] Oct 31 16:01:17.005261 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:17.005229 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-029d26-bbf8b8789-tp5b5"] Oct 31 16:01:18.895983 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:18.895951 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c486ec0f-d3a7-4625-8bae-1dca830110d2" path="/var/lib/kubelet/pods/c486ec0f-d3a7-4625-8bae-1dca830110d2/volumes" Oct 31 16:01:26.206660 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:01:26.206616 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:01:29.165405 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.165380 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc"] Oct 31 16:01:29.166159 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166132 4399 cpu_manager.go:414] "containerMap: removing container" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" Oct 31 16:01:29.166159 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166159 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166179 4399 cpu_manager.go:414] "containerMap: removing container" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerName="cleanup" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166187 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerName="cleanup" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166219 4399 cpu_manager.go:414] "containerMap: removing container" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166225 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166235 4399 cpu_manager.go:414] "containerMap: removing container" podUID="c486ec0f-d3a7-4625-8bae-1dca830110d2" containerName="payload-processing" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166239 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c486ec0f-d3a7-4625-8bae-1dca830110d2" containerName="payload-processing" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166247 4399 cpu_manager.go:414] "containerMap: removing container" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166251 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83e1c67-842d-4759-8e56-7d14d56f4e51" containerName="istio-proxy" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166261 4399 cpu_manager.go:414] "containerMap: removing container" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166265 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166283 4399 cpu_manager.go:414] "containerMap: removing container" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerName="cleanup" Oct 31 16:01:29.166370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166287 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerName="cleanup" Oct 31 16:01:29.166864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166393 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" Oct 31 16:01:29.166864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166408 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerName="cleanup" Oct 31 16:01:29.166864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166415 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerName="cleanup" Oct 31 16:01:29.166864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166423 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f4b118-d329-4758-99d6-87367efd0468" containerName="istio-proxy" Oct 31 16:01:29.166864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166428 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerName="cleanup" Oct 31 16:01:29.166864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.166437 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c486ec0f-d3a7-4625-8bae-1dca830110d2" containerName="payload-processing" Oct 31 16:01:29.171303 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.171285 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.251156 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.251119 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.251305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.251160 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.251305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.251232 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.251401 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.251329 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.251401 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.251349 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.251509 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.251462 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c50f0a4-9681-40dd-bce2-c2026a73b373-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.251509 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.251487 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.251607 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.251576 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.251659 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.251623 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpf4\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-kube-api-access-8dpf4\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.274789 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.274754 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc"] Oct 31 16:01:29.311567 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.311535 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-029d26-openshift-default-dockercfg-lxm75\"" Oct 31 16:01:29.352352 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352322 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.352506 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352365 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpf4\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-kube-api-access-8dpf4\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.352506 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352406 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.352506 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352435 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.352506 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352450 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.352730 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352677 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.352855 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352758 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.352900 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352849 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-data\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.352900 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352872 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c50f0a4-9681-40dd-bce2-c2026a73b373-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.353010 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352908 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.353010 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.352917 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.353128 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.353056 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-credential-socket\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.353194 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.353126 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-certs\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.353519 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.353494 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c50f0a4-9681-40dd-bce2-c2026a73b373-istiod-ca-cert\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.355167 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.355141 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-podinfo\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.355222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.355170 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-envoy\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.409174 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.409150 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-token\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.409335 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.409318 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpf4\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-kube-api-access-8dpf4\") pod \"e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.484971 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.484944 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:29.915543 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.915520 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc"] Oct 31 16:01:29.917026 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:01:29.917000 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c50f0a4_9681_40dd_bce2_c2026a73b373.slice/crio-cf549b28c4fc8562952ad9ece0868a92f2bfeb7ae2914c5a4d071ffe507c9f99 WatchSource:0}: Error finding container cf549b28c4fc8562952ad9ece0868a92f2bfeb7ae2914c5a4d071ffe507c9f99: Status 404 returned error can't find the container with id cf549b28c4fc8562952ad9ece0868a92f2bfeb7ae2914c5a4d071ffe507c9f99 Oct 31 16:01:29.919117 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.919085 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:01:29.919198 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.919148 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:01:29.919198 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:29.919182 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:01:30.895916 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:30.895876 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" event={"ID":"1c50f0a4-9681-40dd-bce2-c2026a73b373","Type":"ContainerStarted","Data":"2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada"} Oct 31 16:01:30.896305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:30.895938 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" event={"ID":"1c50f0a4-9681-40dd-bce2-c2026a73b373","Type":"ContainerStarted","Data":"cf549b28c4fc8562952ad9ece0868a92f2bfeb7ae2914c5a4d071ffe507c9f99"} Oct 31 16:01:31.100274 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:31.100222 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" podStartSLOduration=2.100208194 podStartE2EDuration="2.100208194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:01:31.098806736 +0000 UTC m=+4534.310048150" watchObservedRunningTime="2026-10-31 16:01:31.100208194 +0000 UTC m=+4534.311449606" Oct 31 16:01:31.486054 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:31.486019 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:31.487772 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:31.487746 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 16:01:31.487922 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:31.487803 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 16:01:32.485856 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:32.485819 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 16:01:32.486278 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:32.485886 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 16:01:33.486093 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:33.486045 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 16:01:33.486548 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:33.486121 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 16:01:34.486048 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:34.486008 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 16:01:34.486239 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:34.486089 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 16:01:35.490115 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:35.490076 4399 patch_prober.go:29] interesting pod/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 31 16:01:35.490548 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:35.490145 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 31 16:01:36.504736 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:36.502449 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:36.504736 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:36.504352 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:36.507845 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:36.507810 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:01:54.626772 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:54.626736 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc"] Oct 31 16:01:54.627334 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:54.626936 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerName="istio-proxy" containerID="cri-o://2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada" gracePeriod=30 Oct 31 16:01:56.207402 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:01:56.207365 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:01:59.905838 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:01:59.905770 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:02:00.000987 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.000956 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-podinfo\") pod \"1c50f0a4-9681-40dd-bce2-c2026a73b373\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " Oct 31 16:02:00.001189 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001003 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-certs\") pod \"1c50f0a4-9681-40dd-bce2-c2026a73b373\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " Oct 31 16:02:00.001189 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001046 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c50f0a4-9681-40dd-bce2-c2026a73b373-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c50f0a4-9681-40dd-bce2-c2026a73b373-istiod-ca-cert\") pod \"1c50f0a4-9681-40dd-bce2-c2026a73b373\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " Oct 31 16:02:00.001189 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001170 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-kube-api-access-8dpf4\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-kube-api-access-8dpf4\") pod \"1c50f0a4-9681-40dd-bce2-c2026a73b373\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " Oct 31 16:02:00.001368 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001228 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-data\") pod \"1c50f0a4-9681-40dd-bce2-c2026a73b373\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " Oct 31 16:02:00.001368 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001296 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-certs" pod "1c50f0a4-9681-40dd-bce2-c2026a73b373" (UID: "1c50f0a4-9681-40dd-bce2-c2026a73b373"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:02:00.001368 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001306 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-envoy\") pod \"1c50f0a4-9681-40dd-bce2-c2026a73b373\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " Oct 31 16:02:00.001539 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001397 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c50f0a4-9681-40dd-bce2-c2026a73b373-istiod-ca-cert" pod "1c50f0a4-9681-40dd-bce2-c2026a73b373" (UID: "1c50f0a4-9681-40dd-bce2-c2026a73b373"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:02:00.001539 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001531 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-token\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-token\") pod \"1c50f0a4-9681-40dd-bce2-c2026a73b373\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " Oct 31 16:02:00.001639 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001581 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-data" pod "1c50f0a4-9681-40dd-bce2-c2026a73b373" (UID: "1c50f0a4-9681-40dd-bce2-c2026a73b373"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:02:00.001639 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001587 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-socket\") pod \"1c50f0a4-9681-40dd-bce2-c2026a73b373\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " Oct 31 16:02:00.001792 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001639 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-credential-socket\") pod \"1c50f0a4-9681-40dd-bce2-c2026a73b373\" (UID: \"1c50f0a4-9681-40dd-bce2-c2026a73b373\") " Oct 31 16:02:00.002017 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.001943 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-credential-socket" pod "1c50f0a4-9681-40dd-bce2-c2026a73b373" (UID: "1c50f0a4-9681-40dd-bce2-c2026a73b373"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:02:00.002271 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.002199 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.002271 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.002222 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.002271 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.002219 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-socket" pod "1c50f0a4-9681-40dd-bce2-c2026a73b373" (UID: "1c50f0a4-9681-40dd-bce2-c2026a73b373"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:02:00.002271 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.002235 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c50f0a4-9681-40dd-bce2-c2026a73b373-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.002271 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.002249 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.003623 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.003595 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-podinfo" pod "1c50f0a4-9681-40dd-bce2-c2026a73b373" (UID: "1c50f0a4-9681-40dd-bce2-c2026a73b373"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:02:00.003766 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.003724 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-envoy" pod "1c50f0a4-9681-40dd-bce2-c2026a73b373" (UID: "1c50f0a4-9681-40dd-bce2-c2026a73b373"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:02:00.004086 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.004063 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-kube-api-access-8dpf4" pod "1c50f0a4-9681-40dd-bce2-c2026a73b373" (UID: "1c50f0a4-9681-40dd-bce2-c2026a73b373"). InnerVolumeSpecName "kube-api-access-8dpf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:02:00.004322 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.004304 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-token" pod "1c50f0a4-9681-40dd-bce2-c2026a73b373" (UID: "1c50f0a4-9681-40dd-bce2-c2026a73b373"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:02:00.045240 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.045213 4399 generic.go:356] "Generic (PLEG): container finished" podID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerID="2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada" exitCode=0 Oct 31 16:02:00.045389 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.045258 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" Oct 31 16:02:00.045389 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.045297 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" event={"ID":"1c50f0a4-9681-40dd-bce2-c2026a73b373","Type":"ContainerDied","Data":"2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada"} Oct 31 16:02:00.045389 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.045381 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc" event={"ID":"1c50f0a4-9681-40dd-bce2-c2026a73b373","Type":"ContainerDied","Data":"cf549b28c4fc8562952ad9ece0868a92f2bfeb7ae2914c5a4d071ffe507c9f99"} Oct 31 16:02:00.045516 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.045402 4399 scope.go:122] "RemoveContainer" containerID="2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada" Oct 31 16:02:00.056481 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.056462 4399 scope.go:122] "RemoveContainer" containerID="2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada" Oct 31 16:02:00.056923 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:02:00.056844 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada\": container with ID starting with 2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada not found: ID does not exist" containerID="2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada" Oct 31 16:02:00.056923 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.056892 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada"} err="failed to get container status \"2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada\": rpc error: code = NotFound desc = could not find container \"2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada\": container with ID starting with 2a903f0b978ca5cd84caf2da9ee9fdebe5ffe7f9772b83052166110398010ada not found: ID does not exist" Oct 31 16:02:00.103616 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.103584 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.103616 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.103618 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpf4\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-kube-api-access-8dpf4\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.103616 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.103633 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.103942 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.103645 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c50f0a4-9681-40dd-bce2-c2026a73b373-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.103942 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.103654 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c50f0a4-9681-40dd-bce2-c2026a73b373-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:02:00.231068 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.231036 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc"] Oct 31 16:02:00.235109 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.235086 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-029d26-openshift-default-74cc46d5f9-2gdzc"] Oct 31 16:02:00.895891 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:02:00.895863 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" path="/var/lib/kubelet/pods/1c50f0a4-9681-40dd-bce2-c2026a73b373/volumes" Oct 31 16:02:26.208445 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:02:26.208399 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:02:56.206795 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:02:56.206762 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:03:26.206394 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:03:26.206351 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:03:56.206227 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:03:56.206181 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:04:03.462561 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.462528 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p"] Oct 31 16:04:03.463017 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.462773 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" podUID="2f56fdda-d597-4e66-9e41-77092b8f99d9" containerName="maas-api" containerID="cri-o://a17d5bae9d27ec4b044641396c99fee0b41b6b00656c8914d51d02814d06e85b" gracePeriod=30 Oct 31 16:04:03.721176 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.721059 4399 generic.go:356] "Generic (PLEG): container finished" podID="2f56fdda-d597-4e66-9e41-77092b8f99d9" containerID="a17d5bae9d27ec4b044641396c99fee0b41b6b00656c8914d51d02814d06e85b" exitCode=0 Oct 31 16:04:03.721176 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.721143 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" event={"ID":"2f56fdda-d597-4e66-9e41-77092b8f99d9","Type":"ContainerDied","Data":"a17d5bae9d27ec4b044641396c99fee0b41b6b00656c8914d51d02814d06e85b"} Oct 31 16:04:03.745733 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.745685 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 16:04:03.800965 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.800941 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f56fdda-d597-4e66-9e41-77092b8f99d9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f56fdda-d597-4e66-9e41-77092b8f99d9-openshift-service-ca\") pod \"2f56fdda-d597-4e66-9e41-77092b8f99d9\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " Oct 31 16:04:03.801099 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.801011 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-metrics-certs\") pod \"2f56fdda-d597-4e66-9e41-77092b8f99d9\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " Oct 31 16:04:03.801099 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.801034 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f56fdda-d597-4e66-9e41-77092b8f99d9-kube-api-access-bjfk9\" (UniqueName: \"kubernetes.io/projected/2f56fdda-d597-4e66-9e41-77092b8f99d9-kube-api-access-bjfk9\") pod \"2f56fdda-d597-4e66-9e41-77092b8f99d9\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " Oct 31 16:04:03.801099 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.801097 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls\") pod \"2f56fdda-d597-4e66-9e41-77092b8f99d9\" (UID: \"2f56fdda-d597-4e66-9e41-77092b8f99d9\") " Oct 31 16:04:03.801288 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.801266 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f56fdda-d597-4e66-9e41-77092b8f99d9-openshift-service-ca" pod "2f56fdda-d597-4e66-9e41-77092b8f99d9" (UID: "2f56fdda-d597-4e66-9e41-77092b8f99d9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:03.801510 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.801489 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f56fdda-d597-4e66-9e41-77092b8f99d9-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:03.803350 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.803325 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-metrics-certs" pod "2f56fdda-d597-4e66-9e41-77092b8f99d9" (UID: "2f56fdda-d597-4e66-9e41-77092b8f99d9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:04:03.803350 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.803333 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f56fdda-d597-4e66-9e41-77092b8f99d9-kube-api-access-bjfk9" pod "2f56fdda-d597-4e66-9e41-77092b8f99d9" (UID: "2f56fdda-d597-4e66-9e41-77092b8f99d9"). InnerVolumeSpecName "kube-api-access-bjfk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:03.803479 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.803346 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls" pod "2f56fdda-d597-4e66-9e41-77092b8f99d9" (UID: "2f56fdda-d597-4e66-9e41-77092b8f99d9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:04:03.902521 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.902488 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:03.902521 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.902520 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f56fdda-d597-4e66-9e41-77092b8f99d9-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:03.902792 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:03.902536 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfk9\" (UniqueName: \"kubernetes.io/projected/2f56fdda-d597-4e66-9e41-77092b8f99d9-kube-api-access-bjfk9\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:04.727902 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:04.727873 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" Oct 31 16:04:04.728358 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:04.727909 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p" event={"ID":"2f56fdda-d597-4e66-9e41-77092b8f99d9","Type":"ContainerDied","Data":"73d5312a2e4c0f84f3f49cec55b6226e1efacef4f202d6d343b8dbe5e9af26f0"} Oct 31 16:04:04.728358 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:04.727962 4399 scope.go:122] "RemoveContainer" containerID="a17d5bae9d27ec4b044641396c99fee0b41b6b00656c8914d51d02814d06e85b" Oct 31 16:04:04.896720 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:04.896663 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p"] Oct 31 16:04:04.939866 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:04.939837 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-531c3c-6cbc589588-t7d2p"] Oct 31 16:04:06.895517 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:06.895484 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f56fdda-d597-4e66-9e41-77092b8f99d9" path="/var/lib/kubelet/pods/2f56fdda-d597-4e66-9e41-77092b8f99d9/volumes" Oct 31 16:04:20.405838 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:20.405797 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8"] Oct 31 16:04:20.406333 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:20.406025 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" containerID="cri-o://3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f" gracePeriod=30 Oct 31 16:04:20.456184 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:20.456138 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm"] Oct 31 16:04:20.456411 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:20.456385 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" containerID="cri-o://e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac" gracePeriod=30 Oct 31 16:04:25.731728 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.731705 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 16:04:25.753782 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.753752 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 16:04:25.771435 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771408 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-token\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-token\") pod \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " Oct 31 16:04:25.771571 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771444 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-data\") pod \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " Oct 31 16:04:25.771571 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771464 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-data\") pod \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " Oct 31 16:04:25.771571 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771493 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-kube-api-access-skdzg\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-kube-api-access-skdzg\") pod \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " Oct 31 16:04:25.771571 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771522 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-envoy\") pod \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " Oct 31 16:04:25.771571 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771559 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-credential-socket\") pod \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " Oct 31 16:04:25.771878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771608 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istiod-ca-cert\") pod \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " Oct 31 16:04:25.771878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771643 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-socket\") pod \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " Oct 31 16:04:25.771878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771674 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-kube-api-access-r5wjx\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-kube-api-access-r5wjx\") pod \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " Oct 31 16:04:25.771878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771735 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-envoy\") pod \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " Oct 31 16:04:25.771878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771762 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-token\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-token\") pod \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " Oct 31 16:04:25.771878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771781 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-data" pod "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" (UID: "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.771878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771814 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-data" pod "3c9ff191-9669-4e72-b06d-6b1518c1dea1" (UID: "3c9ff191-9669-4e72-b06d-6b1518c1dea1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.771878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771799 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-podinfo\") pod \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " Oct 31 16:04:25.772305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771871 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-credential-socket" pod "3c9ff191-9669-4e72-b06d-6b1518c1dea1" (UID: "3c9ff191-9669-4e72-b06d-6b1518c1dea1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.772305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771900 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-podinfo\") pod \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " Oct 31 16:04:25.772305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771960 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-socket\") pod \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " Oct 31 16:04:25.772305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.771995 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-certs\") pod \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " Oct 31 16:04:25.772305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.772055 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istiod-ca-cert\") pod \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " Oct 31 16:04:25.772305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.772142 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-credential-socket\") pod \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\" (UID: \"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da\") " Oct 31 16:04:25.772305 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.772202 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-certs\") pod \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\" (UID: \"3c9ff191-9669-4e72-b06d-6b1518c1dea1\") " Oct 31 16:04:25.772657 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.772636 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.772750 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.772654 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.772750 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.772669 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.772899 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.772877 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-certs" pod "3c9ff191-9669-4e72-b06d-6b1518c1dea1" (UID: "3c9ff191-9669-4e72-b06d-6b1518c1dea1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.773072 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.773043 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-socket" pod "3c9ff191-9669-4e72-b06d-6b1518c1dea1" (UID: "3c9ff191-9669-4e72-b06d-6b1518c1dea1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.773478 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.773455 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-certs" pod "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" (UID: "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.773911 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.773890 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istiod-ca-cert" pod "3c9ff191-9669-4e72-b06d-6b1518c1dea1" (UID: "3c9ff191-9669-4e72-b06d-6b1518c1dea1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:25.774310 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.774281 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-credential-socket" pod "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" (UID: "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.774412 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.774373 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-envoy" pod "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" (UID: "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.774514 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.774489 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-token" pod "3c9ff191-9669-4e72-b06d-6b1518c1dea1" (UID: "3c9ff191-9669-4e72-b06d-6b1518c1dea1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:25.774635 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.774608 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istiod-ca-cert" pod "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" (UID: "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:25.774756 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.774735 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-socket" pod "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" (UID: "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.774816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.774733 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-podinfo" pod "3c9ff191-9669-4e72-b06d-6b1518c1dea1" (UID: "3c9ff191-9669-4e72-b06d-6b1518c1dea1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:04:25.774816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.774792 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-kube-api-access-skdzg" pod "3c9ff191-9669-4e72-b06d-6b1518c1dea1" (UID: "3c9ff191-9669-4e72-b06d-6b1518c1dea1"). InnerVolumeSpecName "kube-api-access-skdzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:25.775019 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.774997 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-kube-api-access-r5wjx" pod "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" (UID: "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da"). InnerVolumeSpecName "kube-api-access-r5wjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:25.775199 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.775038 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-envoy" pod "3c9ff191-9669-4e72-b06d-6b1518c1dea1" (UID: "3c9ff191-9669-4e72-b06d-6b1518c1dea1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:04:25.775378 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.775358 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-podinfo" pod "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" (UID: "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:04:25.776487 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.776463 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-token" pod "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" (UID: "bf2c4a7a-c75b-40cc-b6bb-2af983ef67da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:25.840530 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.840508 4399 generic.go:356] "Generic (PLEG): container finished" podID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerID="3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f" exitCode=0 Oct 31 16:04:25.840629 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.840563 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" Oct 31 16:04:25.840629 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.840602 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" event={"ID":"3c9ff191-9669-4e72-b06d-6b1518c1dea1","Type":"ContainerDied","Data":"3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f"} Oct 31 16:04:25.840753 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.840681 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8" event={"ID":"3c9ff191-9669-4e72-b06d-6b1518c1dea1","Type":"ContainerDied","Data":"0f4bf0fea12058b3b37ab9cfd372012562ac3b0911be181d04ce1ea99fe529ed"} Oct 31 16:04:25.840753 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.840727 4399 scope.go:122] "RemoveContainer" containerID="3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f" Oct 31 16:04:25.842052 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.842027 4399 generic.go:356] "Generic (PLEG): container finished" podID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerID="e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac" exitCode=0 Oct 31 16:04:25.842162 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.842064 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" Oct 31 16:04:25.842162 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.842100 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" event={"ID":"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da","Type":"ContainerDied","Data":"e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac"} Oct 31 16:04:25.842162 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.842160 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm" event={"ID":"bf2c4a7a-c75b-40cc-b6bb-2af983ef67da","Type":"ContainerDied","Data":"9ff9e4a68865e77ee5491f1195e0e1f1772d0d83e3fcda5a58ea2c4e0f0e5278"} Oct 31 16:04:25.851516 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.851496 4399 scope.go:122] "RemoveContainer" containerID="3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f" Oct 31 16:04:25.851846 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:04:25.851818 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f\": container with ID starting with 3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f not found: ID does not exist" containerID="3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f" Oct 31 16:04:25.851955 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.851854 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f"} err="failed to get container status \"3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f\": rpc error: code = NotFound desc = could not find container \"3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f\": container with ID starting with 3dc2f6511d8e76674462db705322cc40ad05c075da7dd8f4d5cac23a55cb571f not found: ID does not exist" Oct 31 16:04:25.851955 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.851876 4399 scope.go:122] "RemoveContainer" containerID="e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac" Oct 31 16:04:25.861616 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.861591 4399 scope.go:122] "RemoveContainer" containerID="e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac" Oct 31 16:04:25.861905 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:04:25.861887 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac\": container with ID starting with e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac not found: ID does not exist" containerID="e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac" Oct 31 16:04:25.861985 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.861912 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac"} err="failed to get container status \"e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac\": rpc error: code = NotFound desc = could not find container \"e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac\": container with ID starting with e159e3dc48125a7605b08fcc90192555cdb2854068536467a4e39ef1181e86ac not found: ID does not exist" Oct 31 16:04:25.873406 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873384 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873406 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873408 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873418 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873426 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873436 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873444 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873452 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdzg\" (UniqueName: \"kubernetes.io/projected/3c9ff191-9669-4e72-b06d-6b1518c1dea1-kube-api-access-skdzg\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873461 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873469 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873476 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873485 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5wjx\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-kube-api-access-r5wjx\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873493 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873500 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873509 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c9ff191-9669-4e72-b06d-6b1518c1dea1-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:25.873527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:25.873517 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:26.006734 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:26.006599 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8"] Oct 31 16:04:26.010385 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:26.010356 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-97lt8"] Oct 31 16:04:26.180597 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:26.180563 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm"] Oct 31 16:04:26.184065 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:26.184044 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pkkhm"] Oct 31 16:04:26.206908 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:04:26.206882 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:04:26.894995 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:26.894964 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" path="/var/lib/kubelet/pods/3c9ff191-9669-4e72-b06d-6b1518c1dea1/volumes" Oct 31 16:04:26.895404 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:26.895393 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" path="/var/lib/kubelet/pods/bf2c4a7a-c75b-40cc-b6bb-2af983ef67da/volumes" Oct 31 16:04:47.400362 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.400269 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn"] Oct 31 16:04:47.400845 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.400513 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" podUID="e65e4c6b-bd54-420c-b113-fb3ead82543b" containerName="maas-api" containerID="cri-o://fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da" gracePeriod=30 Oct 31 16:04:47.522808 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.522777 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" podUID="e65e4c6b-bd54-420c-b113-fb3ead82543b" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.129:8443/health\": dial tcp 10.133.0.129:8443: connect: connection refused" Oct 31 16:04:47.669769 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.669747 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 16:04:47.829039 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.829009 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65e4c6b-bd54-420c-b113-fb3ead82543b-kube-api-access-vq862\" (UniqueName: \"kubernetes.io/projected/e65e4c6b-bd54-420c-b113-fb3ead82543b-kube-api-access-vq862\") pod \"e65e4c6b-bd54-420c-b113-fb3ead82543b\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " Oct 31 16:04:47.829214 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.829079 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65e4c6b-bd54-420c-b113-fb3ead82543b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65e4c6b-bd54-420c-b113-fb3ead82543b-openshift-service-ca\") pod \"e65e4c6b-bd54-420c-b113-fb3ead82543b\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " Oct 31 16:04:47.829214 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.829143 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-metrics-certs\") pod \"e65e4c6b-bd54-420c-b113-fb3ead82543b\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " Oct 31 16:04:47.829214 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.829194 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls\") pod \"e65e4c6b-bd54-420c-b113-fb3ead82543b\" (UID: \"e65e4c6b-bd54-420c-b113-fb3ead82543b\") " Oct 31 16:04:47.829491 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.829466 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65e4c6b-bd54-420c-b113-fb3ead82543b-openshift-service-ca" pod "e65e4c6b-bd54-420c-b113-fb3ead82543b" (UID: "e65e4c6b-bd54-420c-b113-fb3ead82543b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:04:47.829664 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.829646 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e65e4c6b-bd54-420c-b113-fb3ead82543b-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:47.831587 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.831489 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-metrics-certs" pod "e65e4c6b-bd54-420c-b113-fb3ead82543b" (UID: "e65e4c6b-bd54-420c-b113-fb3ead82543b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:04:47.831587 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.831495 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65e4c6b-bd54-420c-b113-fb3ead82543b-kube-api-access-vq862" pod "e65e4c6b-bd54-420c-b113-fb3ead82543b" (UID: "e65e4c6b-bd54-420c-b113-fb3ead82543b"). InnerVolumeSpecName "kube-api-access-vq862". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:04:47.831750 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.831581 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls" pod "e65e4c6b-bd54-420c-b113-fb3ead82543b" (UID: "e65e4c6b-bd54-420c-b113-fb3ead82543b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:04:47.930672 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.930602 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq862\" (UniqueName: \"kubernetes.io/projected/e65e4c6b-bd54-420c-b113-fb3ead82543b-kube-api-access-vq862\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:47.930672 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.930628 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:47.930672 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.930637 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e65e4c6b-bd54-420c-b113-fb3ead82543b-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:04:47.990415 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.990376 4399 generic.go:356] "Generic (PLEG): container finished" podID="e65e4c6b-bd54-420c-b113-fb3ead82543b" containerID="fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da" exitCode=0 Oct 31 16:04:47.990588 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.990447 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" Oct 31 16:04:47.990588 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.990462 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" event={"ID":"e65e4c6b-bd54-420c-b113-fb3ead82543b","Type":"ContainerDied","Data":"fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da"} Oct 31 16:04:47.990588 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.990544 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn" event={"ID":"e65e4c6b-bd54-420c-b113-fb3ead82543b","Type":"ContainerDied","Data":"1cdbaa24b5637a55af1d53421d658ed92b7a2a800a92b9e502e677087938900d"} Oct 31 16:04:47.990588 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:47.990570 4399 scope.go:122] "RemoveContainer" containerID="fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da" Oct 31 16:04:48.000690 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:48.000672 4399 scope.go:122] "RemoveContainer" containerID="fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da" Oct 31 16:04:48.000948 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:04:48.000932 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da\": container with ID starting with fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da not found: ID does not exist" containerID="fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da" Oct 31 16:04:48.000998 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:48.000956 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da"} err="failed to get container status \"fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da\": rpc error: code = NotFound desc = could not find container \"fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da\": container with ID starting with fbbe5b86d94aaddf63e47b0218a1565c89ee07b144e4cc8922f6fd987df3a0da not found: ID does not exist" Oct 31 16:04:48.153157 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:48.153129 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn"] Oct 31 16:04:48.202753 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:48.202722 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-55b3fb-768ddb86-qrgdn"] Oct 31 16:04:48.895856 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:04:48.895824 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65e4c6b-bd54-420c-b113-fb3ead82543b" path="/var/lib/kubelet/pods/e65e4c6b-bd54-420c-b113-fb3ead82543b/volumes" Oct 31 16:04:56.207729 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:04:56.207664 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:05:00.944278 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:00.944243 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c"] Oct 31 16:05:00.993627 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:00.993599 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j"] Oct 31 16:05:15.966447 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:15.966395 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" containerID="cri-o://83515f58d50e72661c5e3cd40aa009a4e684fc5897658a89ed64dc758237db02" gracePeriod=45 Oct 31 16:05:16.014756 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.014691 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" containerID="cri-o://a3bb92da2f0d4008f9d65d51c51771f3df04946049241dc6321226f0c23d35af" gracePeriod=45 Oct 31 16:05:16.153794 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.153446 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/ready\": dial tcp 10.133.0.70:8000: connect: connection refused" Oct 31 16:05:16.160296 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.160124 4399 generic.go:356] "Generic (PLEG): container finished" podID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerID="a3bb92da2f0d4008f9d65d51c51771f3df04946049241dc6321226f0c23d35af" exitCode=0 Oct 31 16:05:16.160296 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.160231 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" event={"ID":"b5d10edc-610e-4f43-8676-ab2e687db74b","Type":"ContainerDied","Data":"a3bb92da2f0d4008f9d65d51c51771f3df04946049241dc6321226f0c23d35af"} Oct 31 16:05:16.162867 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.162653 4399 generic.go:356] "Generic (PLEG): container finished" podID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerID="83515f58d50e72661c5e3cd40aa009a4e684fc5897658a89ed64dc758237db02" exitCode=0 Oct 31 16:05:16.162867 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.162821 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" event={"ID":"14193ed6-7ca5-442b-90d7-b5baff761c16","Type":"ContainerDied","Data":"83515f58d50e72661c5e3cd40aa009a4e684fc5897658a89ed64dc758237db02"} Oct 31 16:05:16.359739 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.348860 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 16:05:16.359739 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.355947 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 16:05:16.433017 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.432982 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d10edc-610e-4f43-8676-ab2e687db74b-kube-api-access-mkbh4\" (UniqueName: \"kubernetes.io/projected/b5d10edc-610e-4f43-8676-ab2e687db74b-kube-api-access-mkbh4\") pod \"b5d10edc-610e-4f43-8676-ab2e687db74b\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " Oct 31 16:05:16.433212 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433037 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14193ed6-7ca5-442b-90d7-b5baff761c16-tls-certs\" (UniqueName: \"kubernetes.io/secret/14193ed6-7ca5-442b-90d7-b5baff761c16-tls-certs\") pod \"14193ed6-7ca5-442b-90d7-b5baff761c16\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " Oct 31 16:05:16.433212 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433089 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-dshm\") pod \"b5d10edc-610e-4f43-8676-ab2e687db74b\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " Oct 31 16:05:16.433212 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433119 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-home\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-home\") pod \"14193ed6-7ca5-442b-90d7-b5baff761c16\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " Oct 31 16:05:16.433399 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433346 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-home" pod "14193ed6-7ca5-442b-90d7-b5baff761c16" (UID: "14193ed6-7ca5-442b-90d7-b5baff761c16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.433399 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433366 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-model-cache\") pod \"14193ed6-7ca5-442b-90d7-b5baff761c16\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " Oct 31 16:05:16.433528 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433484 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14193ed6-7ca5-442b-90d7-b5baff761c16-kube-api-access-5r4vj\" (UniqueName: \"kubernetes.io/projected/14193ed6-7ca5-442b-90d7-b5baff761c16-kube-api-access-5r4vj\") pod \"14193ed6-7ca5-442b-90d7-b5baff761c16\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " Oct 31 16:05:16.433528 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433515 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-tmp-dir\") pod \"b5d10edc-610e-4f43-8676-ab2e687db74b\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " Oct 31 16:05:16.433833 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433582 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5d10edc-610e-4f43-8676-ab2e687db74b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d10edc-610e-4f43-8676-ab2e687db74b-tls-certs\") pod \"b5d10edc-610e-4f43-8676-ab2e687db74b\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " Oct 31 16:05:16.433833 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433605 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-model-cache\") pod \"b5d10edc-610e-4f43-8676-ab2e687db74b\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " Oct 31 16:05:16.433833 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433622 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-tmp-dir\") pod \"14193ed6-7ca5-442b-90d7-b5baff761c16\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " Oct 31 16:05:16.433833 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433648 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-dshm\") pod \"14193ed6-7ca5-442b-90d7-b5baff761c16\" (UID: \"14193ed6-7ca5-442b-90d7-b5baff761c16\") " Oct 31 16:05:16.433833 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433664 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-home\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-home\") pod \"b5d10edc-610e-4f43-8676-ab2e687db74b\" (UID: \"b5d10edc-610e-4f43-8676-ab2e687db74b\") " Oct 31 16:05:16.434139 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.434079 4399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-home\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.434139 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.433659 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-model-cache" pod "14193ed6-7ca5-442b-90d7-b5baff761c16" (UID: "14193ed6-7ca5-442b-90d7-b5baff761c16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.434139 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.434080 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-model-cache" pod "b5d10edc-610e-4f43-8676-ab2e687db74b" (UID: "b5d10edc-610e-4f43-8676-ab2e687db74b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.434435 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.434399 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-tmp-dir" pod "14193ed6-7ca5-442b-90d7-b5baff761c16" (UID: "14193ed6-7ca5-442b-90d7-b5baff761c16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.434435 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.434425 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-tmp-dir" pod "b5d10edc-610e-4f43-8676-ab2e687db74b" (UID: "b5d10edc-610e-4f43-8676-ab2e687db74b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.434644 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.434477 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-home" pod "b5d10edc-610e-4f43-8676-ab2e687db74b" (UID: "b5d10edc-610e-4f43-8676-ab2e687db74b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.435896 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.435868 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14193ed6-7ca5-442b-90d7-b5baff761c16-tls-certs" pod "14193ed6-7ca5-442b-90d7-b5baff761c16" (UID: "14193ed6-7ca5-442b-90d7-b5baff761c16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:16.436024 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.435910 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-dshm" pod "b5d10edc-610e-4f43-8676-ab2e687db74b" (UID: "b5d10edc-610e-4f43-8676-ab2e687db74b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.437030 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.436982 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14193ed6-7ca5-442b-90d7-b5baff761c16-kube-api-access-5r4vj" pod "14193ed6-7ca5-442b-90d7-b5baff761c16" (UID: "14193ed6-7ca5-442b-90d7-b5baff761c16"). InnerVolumeSpecName "kube-api-access-5r4vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:16.437192 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.437040 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-dshm" pod "14193ed6-7ca5-442b-90d7-b5baff761c16" (UID: "14193ed6-7ca5-442b-90d7-b5baff761c16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:16.437820 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.437765 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d10edc-610e-4f43-8676-ab2e687db74b-kube-api-access-mkbh4" pod "b5d10edc-610e-4f43-8676-ab2e687db74b" (UID: "b5d10edc-610e-4f43-8676-ab2e687db74b"). InnerVolumeSpecName "kube-api-access-mkbh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:16.438520 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.438499 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5d10edc-610e-4f43-8676-ab2e687db74b-tls-certs" pod "b5d10edc-610e-4f43-8676-ab2e687db74b" (UID: "b5d10edc-610e-4f43-8676-ab2e687db74b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:16.535314 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535239 4399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5d10edc-610e-4f43-8676-ab2e687db74b-tls-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535314 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535267 4399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-model-cache\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535314 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535294 4399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-tmp-dir\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535314 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535303 4399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-dshm\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535314 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535310 4399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-home\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535592 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535319 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkbh4\" (UniqueName: \"kubernetes.io/projected/b5d10edc-610e-4f43-8676-ab2e687db74b-kube-api-access-mkbh4\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535592 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535327 4399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14193ed6-7ca5-442b-90d7-b5baff761c16-tls-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535592 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535335 4399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-dshm\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535592 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535343 4399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14193ed6-7ca5-442b-90d7-b5baff761c16-model-cache\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535592 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535353 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r4vj\" (UniqueName: \"kubernetes.io/projected/14193ed6-7ca5-442b-90d7-b5baff761c16-kube-api-access-5r4vj\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:16.535592 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:16.535362 4399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5d10edc-610e-4f43-8676-ab2e687db74b-tmp-dir\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:17.169192 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.169109 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" Oct 31 16:05:17.169192 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.169123 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j" event={"ID":"b5d10edc-610e-4f43-8676-ab2e687db74b","Type":"ContainerDied","Data":"b73607791a9a6b6f6cccd6d9722e4e45eac306d8e2c437e6d458dc692378a4ca"} Oct 31 16:05:17.169192 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.169174 4399 scope.go:122] "RemoveContainer" containerID="a3bb92da2f0d4008f9d65d51c51771f3df04946049241dc6321226f0c23d35af" Oct 31 16:05:17.170596 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.170577 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" Oct 31 16:05:17.170765 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.170575 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c" event={"ID":"14193ed6-7ca5-442b-90d7-b5baff761c16","Type":"ContainerDied","Data":"7e74ab1e41f50a6d36bcaefe83f97c7c9168207e8bca7195d3fd627b43c6288d"} Oct 31 16:05:17.178148 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.178118 4399 scope.go:122] "RemoveContainer" containerID="83515f58d50e72661c5e3cd40aa009a4e684fc5897658a89ed64dc758237db02" Oct 31 16:05:17.331381 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.331350 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c"] Oct 31 16:05:17.336260 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.336236 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-190693/distinct-w3-190693-kserve-6bc94645d6-45x7c"] Oct 31 16:05:17.443904 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.443819 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j"] Oct 31 16:05:17.445837 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:17.445813 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-190693/simulator-w3-190693-kserve-78c4b4c854-gsn6j"] Oct 31 16:05:18.895471 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:18.895443 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" path="/var/lib/kubelet/pods/14193ed6-7ca5-442b-90d7-b5baff761c16/volumes" Oct 31 16:05:18.895879 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:18.895863 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" path="/var/lib/kubelet/pods/b5d10edc-610e-4f43-8676-ab2e687db74b/volumes" Oct 31 16:05:26.206898 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:05:26.206864 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:05:27.131450 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.131417 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q"] Oct 31 16:05:27.131681 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.131659 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" podUID="3241e52d-5b4e-40eb-b28d-e88c40468457" containerName="maas-api" containerID="cri-o://38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592" gracePeriod=30 Oct 31 16:05:27.402685 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.402662 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 16:05:27.552291 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.552257 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3241e52d-5b4e-40eb-b28d-e88c40468457-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3241e52d-5b4e-40eb-b28d-e88c40468457-openshift-service-ca\") pod \"3241e52d-5b4e-40eb-b28d-e88c40468457\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " Oct 31 16:05:27.552486 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.552363 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3241e52d-5b4e-40eb-b28d-e88c40468457-kube-api-access-kjzns\" (UniqueName: \"kubernetes.io/projected/3241e52d-5b4e-40eb-b28d-e88c40468457-kube-api-access-kjzns\") pod \"3241e52d-5b4e-40eb-b28d-e88c40468457\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " Oct 31 16:05:27.552569 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.552487 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls\") pod \"3241e52d-5b4e-40eb-b28d-e88c40468457\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " Oct 31 16:05:27.552569 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.552523 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-metrics-certs\") pod \"3241e52d-5b4e-40eb-b28d-e88c40468457\" (UID: \"3241e52d-5b4e-40eb-b28d-e88c40468457\") " Oct 31 16:05:27.552689 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.552655 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3241e52d-5b4e-40eb-b28d-e88c40468457-openshift-service-ca" pod "3241e52d-5b4e-40eb-b28d-e88c40468457" (UID: "3241e52d-5b4e-40eb-b28d-e88c40468457"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:27.552883 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.552859 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3241e52d-5b4e-40eb-b28d-e88c40468457-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:27.554818 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.554786 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3241e52d-5b4e-40eb-b28d-e88c40468457-kube-api-access-kjzns" pod "3241e52d-5b4e-40eb-b28d-e88c40468457" (UID: "3241e52d-5b4e-40eb-b28d-e88c40468457"). InnerVolumeSpecName "kube-api-access-kjzns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:27.554818 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.554787 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-metrics-certs" pod "3241e52d-5b4e-40eb-b28d-e88c40468457" (UID: "3241e52d-5b4e-40eb-b28d-e88c40468457"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:27.554965 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.554953 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls" pod "3241e52d-5b4e-40eb-b28d-e88c40468457" (UID: "3241e52d-5b4e-40eb-b28d-e88c40468457"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:27.653919 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.653837 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:27.653919 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.653869 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3241e52d-5b4e-40eb-b28d-e88c40468457-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:27.653919 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:27.653882 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjzns\" (UniqueName: \"kubernetes.io/projected/3241e52d-5b4e-40eb-b28d-e88c40468457-kube-api-access-kjzns\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:28.239975 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.239940 4399 generic.go:356] "Generic (PLEG): container finished" podID="3241e52d-5b4e-40eb-b28d-e88c40468457" containerID="38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592" exitCode=0 Oct 31 16:05:28.240158 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.240000 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" Oct 31 16:05:28.240158 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.240022 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" event={"ID":"3241e52d-5b4e-40eb-b28d-e88c40468457","Type":"ContainerDied","Data":"38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592"} Oct 31 16:05:28.240158 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.240072 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q" event={"ID":"3241e52d-5b4e-40eb-b28d-e88c40468457","Type":"ContainerDied","Data":"2a8ea18bc47ae9056198cab7b3ef5794fc23a4f3da0e0c56ba86dbd949513d59"} Oct 31 16:05:28.240158 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.240094 4399 scope.go:122] "RemoveContainer" containerID="38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592" Oct 31 16:05:28.250573 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.250560 4399 scope.go:122] "RemoveContainer" containerID="38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592" Oct 31 16:05:28.250841 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:05:28.250822 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592\": container with ID starting with 38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592 not found: ID does not exist" containerID="38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592" Oct 31 16:05:28.250912 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.250852 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592"} err="failed to get container status \"38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592\": rpc error: code = NotFound desc = could not find container \"38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592\": container with ID starting with 38658c76776019c55128d51b47c24e3fc4a8d5e38a5d18da8e52f2df71700592 not found: ID does not exist" Oct 31 16:05:28.401884 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.401848 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q"] Oct 31 16:05:28.404662 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.404639 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-e72453-7c85f57858-jr54q"] Oct 31 16:05:28.895926 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:28.895896 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3241e52d-5b4e-40eb-b28d-e88c40468457" path="/var/lib/kubelet/pods/3241e52d-5b4e-40eb-b28d-e88c40468457/volumes" Oct 31 16:05:42.750521 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:42.750490 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g"] Oct 31 16:05:42.751118 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:42.750674 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" podUID="18f3e076-07a1-48a7-9102-273de8d907e4" containerName="maas-api" containerID="cri-o://d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8" gracePeriod=30 Oct 31 16:05:42.797495 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:05:42.797193 4399 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f3e076_07a1_48a7_9102_273de8d907e4.slice/crio-d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8.scope\": RecentStats: unable to find data in memory cache]" Oct 31 16:05:43.023284 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.023260 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 16:05:43.149927 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.149888 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls\") pod \"18f3e076-07a1-48a7-9102-273de8d907e4\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " Oct 31 16:05:43.150124 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.149991 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18f3e076-07a1-48a7-9102-273de8d907e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f3e076-07a1-48a7-9102-273de8d907e4-openshift-service-ca\") pod \"18f3e076-07a1-48a7-9102-273de8d907e4\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " Oct 31 16:05:43.150124 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.150023 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18f3e076-07a1-48a7-9102-273de8d907e4-kube-api-access-99m6t\" (UniqueName: \"kubernetes.io/projected/18f3e076-07a1-48a7-9102-273de8d907e4-kube-api-access-99m6t\") pod \"18f3e076-07a1-48a7-9102-273de8d907e4\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " Oct 31 16:05:43.150124 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.150059 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-metrics-certs\") pod \"18f3e076-07a1-48a7-9102-273de8d907e4\" (UID: \"18f3e076-07a1-48a7-9102-273de8d907e4\") " Oct 31 16:05:43.151021 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.150992 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18f3e076-07a1-48a7-9102-273de8d907e4-openshift-service-ca" pod "18f3e076-07a1-48a7-9102-273de8d907e4" (UID: "18f3e076-07a1-48a7-9102-273de8d907e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:43.152883 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.152854 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-metrics-certs" pod "18f3e076-07a1-48a7-9102-273de8d907e4" (UID: "18f3e076-07a1-48a7-9102-273de8d907e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:43.153539 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.153511 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls" pod "18f3e076-07a1-48a7-9102-273de8d907e4" (UID: "18f3e076-07a1-48a7-9102-273de8d907e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:43.153671 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.153565 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18f3e076-07a1-48a7-9102-273de8d907e4-kube-api-access-99m6t" pod "18f3e076-07a1-48a7-9102-273de8d907e4" (UID: "18f3e076-07a1-48a7-9102-273de8d907e4"). InnerVolumeSpecName "kube-api-access-99m6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:43.256338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.256089 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18f3e076-07a1-48a7-9102-273de8d907e4-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:43.257004 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.256973 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99m6t\" (UniqueName: \"kubernetes.io/projected/18f3e076-07a1-48a7-9102-273de8d907e4-kube-api-access-99m6t\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:43.257141 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.257027 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:43.257141 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.257047 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18f3e076-07a1-48a7-9102-273de8d907e4-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:43.332353 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.332319 4399 generic.go:356] "Generic (PLEG): container finished" podID="18f3e076-07a1-48a7-9102-273de8d907e4" containerID="d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8" exitCode=0 Oct 31 16:05:43.332567 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.332368 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" Oct 31 16:05:43.332567 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.332402 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" event={"ID":"18f3e076-07a1-48a7-9102-273de8d907e4","Type":"ContainerDied","Data":"d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8"} Oct 31 16:05:43.332567 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.332467 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g" event={"ID":"18f3e076-07a1-48a7-9102-273de8d907e4","Type":"ContainerDied","Data":"86dd2fa604673969aa4a61e7680cd02e92becb660d8631b799d72f6a897d84c8"} Oct 31 16:05:43.332567 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.332496 4399 scope.go:122] "RemoveContainer" containerID="d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8" Oct 31 16:05:43.342108 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.342088 4399 scope.go:122] "RemoveContainer" containerID="d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8" Oct 31 16:05:43.342380 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:05:43.342364 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8\": container with ID starting with d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8 not found: ID does not exist" containerID="d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8" Oct 31 16:05:43.342451 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.342387 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8"} err="failed to get container status \"d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8\": rpc error: code = NotFound desc = could not find container \"d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8\": container with ID starting with d39b4e0574f062107b981cdcbef780ea01e8f09a1b74c748632414f67cef26c8 not found: ID does not exist" Oct 31 16:05:43.494017 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.493985 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g"] Oct 31 16:05:43.495819 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:43.495797 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-190693-84fbc9bcf-8gf5g"] Oct 31 16:05:44.896036 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:44.896008 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18f3e076-07a1-48a7-9102-273de8d907e4" path="/var/lib/kubelet/pods/18f3e076-07a1-48a7-9102-273de8d907e4/volumes" Oct 31 16:05:50.635614 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.635575 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22"] Oct 31 16:05:50.636068 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636054 4399 cpu_manager.go:414] "containerMap: removing container" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" Oct 31 16:05:50.636117 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636069 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" Oct 31 16:05:50.636117 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636082 4399 cpu_manager.go:414] "containerMap: removing container" podUID="3241e52d-5b4e-40eb-b28d-e88c40468457" containerName="maas-api" Oct 31 16:05:50.636117 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636087 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3241e52d-5b4e-40eb-b28d-e88c40468457" containerName="maas-api" Oct 31 16:05:50.636117 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636097 4399 cpu_manager.go:414] "containerMap: removing container" podUID="18f3e076-07a1-48a7-9102-273de8d907e4" containerName="maas-api" Oct 31 16:05:50.636117 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636101 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18f3e076-07a1-48a7-9102-273de8d907e4" containerName="maas-api" Oct 31 16:05:50.636117 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636108 4399 cpu_manager.go:414] "containerMap: removing container" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" Oct 31 16:05:50.636117 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636113 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636122 4399 cpu_manager.go:414] "containerMap: removing container" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636128 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636135 4399 cpu_manager.go:414] "containerMap: removing container" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636140 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636148 4399 cpu_manager.go:414] "containerMap: removing container" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerName="istio-proxy" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636152 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerName="istio-proxy" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636162 4399 cpu_manager.go:414] "containerMap: removing container" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerName="cleanup" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636167 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1656a55a-6a22-41ad-b2fa-e550b0d94b3a" containerName="cleanup" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636184 4399 cpu_manager.go:414] "containerMap: removing container" podUID="2f56fdda-d597-4e66-9e41-77092b8f99d9" containerName="maas-api" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636188 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f56fdda-d597-4e66-9e41-77092b8f99d9" containerName="maas-api" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636203 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e65e4c6b-bd54-420c-b113-fb3ead82543b" containerName="maas-api" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636207 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65e4c6b-bd54-420c-b113-fb3ead82543b" containerName="maas-api" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636269 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="14193ed6-7ca5-442b-90d7-b5baff761c16" containerName="main" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636276 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c50f0a4-9681-40dd-bce2-c2026a73b373" containerName="istio-proxy" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636285 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f56fdda-d597-4e66-9e41-77092b8f99d9" containerName="maas-api" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636292 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ab2c02-b50b-44c0-8073-44d25c2b40aa" containerName="istio-proxy" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636300 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65e4c6b-bd54-420c-b113-fb3ead82543b" containerName="maas-api" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636307 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c9ff191-9669-4e72-b06d-6b1518c1dea1" containerName="istio-proxy" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636312 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="3241e52d-5b4e-40eb-b28d-e88c40468457" containerName="maas-api" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636318 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="18f3e076-07a1-48a7-9102-273de8d907e4" containerName="maas-api" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636322 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d10edc-610e-4f43-8676-ab2e687db74b" containerName="main" Oct 31 16:05:50.636338 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.636327 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2c4a7a-c75b-40cc-b6bb-2af983ef67da" containerName="istio-proxy" Oct 31 16:05:50.639623 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.639602 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.691271 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.691247 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-31dff8-openshift-default-dockercfg-5k2pv\"" Oct 31 16:05:50.740818 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.740785 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk885\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-kube-api-access-jk885\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.740969 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.740829 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-certs\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.740969 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.740914 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-data\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.741091 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.740985 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.741091 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.741024 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-token\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.741091 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.741057 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-podinfo\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.741253 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.741094 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-credential-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.741311 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.741259 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istiod-ca-cert\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.741398 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.741382 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-envoy\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.743188 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.743164 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22"] Oct 31 16:05:50.842143 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842104 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-envoy\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842143 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842140 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk885\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-kube-api-access-jk885\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842376 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842158 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-certs\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842376 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842182 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-data\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842376 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842203 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842376 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842316 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-token\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842376 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842354 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-podinfo\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842746 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842385 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-credential-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842746 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842450 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istiod-ca-cert\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842746 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842570 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-certs\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.842900 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.842784 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.843091 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.843066 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-credential-socket\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.843385 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.843332 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-data\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.843385 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.843347 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istiod-ca-cert\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.844736 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.844714 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-envoy\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.845219 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.845200 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-podinfo\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.898584 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.898521 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk885\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-kube-api-access-jk885\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.898971 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.898949 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-token\") pod \"e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:50.953956 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:50.953921 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:51.320082 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:51.319230 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f"] Oct 31 16:05:51.320082 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:51.319464 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" containerID="cri-o://aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa" gracePeriod=30 Oct 31 16:05:51.390003 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:51.389969 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22"] Oct 31 16:05:51.393409 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:05:51.393382 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d96ab8_723c_4e78_9f73_83cbcf01c7b3.slice/crio-64992956862b5af7c1f3186725dbd947a898d7d122f6a26ca25b753f9b688554 WatchSource:0}: Error finding container 64992956862b5af7c1f3186725dbd947a898d7d122f6a26ca25b753f9b688554: Status 404 returned error can't find the container with id 64992956862b5af7c1f3186725dbd947a898d7d122f6a26ca25b753f9b688554 Oct 31 16:05:51.395629 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:51.395598 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:05:51.395753 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:51.395672 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:05:51.395834 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:51.395753 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:05:52.383247 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:52.383189 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" event={"ID":"75d96ab8-723c-4e78-9f73-83cbcf01c7b3","Type":"ContainerStarted","Data":"284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6"} Oct 31 16:05:52.383681 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:52.383264 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" event={"ID":"75d96ab8-723c-4e78-9f73-83cbcf01c7b3","Type":"ContainerStarted","Data":"64992956862b5af7c1f3186725dbd947a898d7d122f6a26ca25b753f9b688554"} Oct 31 16:05:52.965747 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:52.957403 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:52.965747 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:52.957523 4399 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 16:05:52.965747 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:52.957566 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 16:05:53.965843 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:53.965803 4399 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 16:05:53.966266 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:53.965873 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 16:05:54.965761 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:54.963151 4399 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 16:05:54.965761 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:54.963217 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 16:05:55.961080 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:55.961048 4399 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 16:05:55.961339 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:55.961309 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 16:05:56.207294 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:05:56.207261 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:05:56.628993 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.628970 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 16:05:56.716647 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.716616 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istiod-ca-cert\") pod \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " Oct 31 16:05:56.716859 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.716720 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-token\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-token\") pod \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " Oct 31 16:05:56.716859 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.716754 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-certs\") pod \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " Oct 31 16:05:56.716859 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.716790 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-data\") pod \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " Oct 31 16:05:56.716859 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.716844 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-kube-api-access-nfwl7\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-kube-api-access-nfwl7\") pod \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " Oct 31 16:05:56.717110 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.716895 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-socket\") pod \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " Oct 31 16:05:56.717110 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.716935 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-credential-socket\") pod \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " Oct 31 16:05:56.717110 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.716977 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-podinfo\") pod \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " Oct 31 16:05:56.717110 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717016 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-envoy\") pod \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\" (UID: \"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300\") " Oct 31 16:05:56.717110 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717049 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-certs" pod "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" (UID: "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.717448 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717106 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istiod-ca-cert" pod "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" (UID: "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:56.717448 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717129 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-data" pod "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" (UID: "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.717448 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717218 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-socket" pod "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" (UID: "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.717448 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717427 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-credential-socket" pod "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" (UID: "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.717662 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717465 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.717662 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717484 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.717662 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717500 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.717662 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.717514 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.719894 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.719868 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-envoy" pod "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" (UID: "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:05:56.720065 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.720040 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-token" pod "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" (UID: "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:56.720155 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.720139 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-kube-api-access-nfwl7" pod "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" (UID: "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300"). InnerVolumeSpecName "kube-api-access-nfwl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:56.721274 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.721256 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-podinfo" pod "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" (UID: "6207a0ed-36ad-4ef4-aa6a-074f0bc1b300"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:05:56.794221 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.794054 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" podStartSLOduration=6.794037244 podStartE2EDuration="6.794037244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:05:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:05:52.557051718 +0000 UTC m=+4795.768293134" watchObservedRunningTime="2026-10-31 16:05:56.794037244 +0000 UTC m=+4800.005278656" Oct 31 16:05:56.818253 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.818222 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfwl7\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-kube-api-access-nfwl7\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.818253 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.818250 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.818437 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.818260 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.818437 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.818269 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.818437 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.818277 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:56.966917 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.966874 4399 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 31 16:05:56.967100 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:56.966931 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 31 16:05:57.414420 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.414385 4399 generic.go:356] "Generic (PLEG): container finished" podID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerID="aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa" exitCode=0 Oct 31 16:05:57.414905 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.414429 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" Oct 31 16:05:57.414905 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.414471 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" event={"ID":"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300","Type":"ContainerDied","Data":"aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa"} Oct 31 16:05:57.414905 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.414527 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f" event={"ID":"6207a0ed-36ad-4ef4-aa6a-074f0bc1b300","Type":"ContainerDied","Data":"a52fe226b93f28cc26e361f2740ca6ebfc7ee7b34c4af1f44f914013f28aa088"} Oct 31 16:05:57.414905 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.414555 4399 scope.go:122] "RemoveContainer" containerID="aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa" Oct 31 16:05:57.423690 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.423672 4399 scope.go:122] "RemoveContainer" containerID="f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0" Oct 31 16:05:57.431905 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.431889 4399 scope.go:122] "RemoveContainer" containerID="aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa" Oct 31 16:05:57.432185 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:05:57.432161 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa\": container with ID starting with aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa not found: ID does not exist" containerID="aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa" Oct 31 16:05:57.432257 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.432195 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa"} err="failed to get container status \"aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa\": rpc error: code = NotFound desc = could not find container \"aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa\": container with ID starting with aad6d6f5d9f32762216a178137ed0c152702c5456b03021225be0588810ababa not found: ID does not exist" Oct 31 16:05:57.432257 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.432214 4399 scope.go:122] "RemoveContainer" containerID="f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0" Oct 31 16:05:57.432508 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:05:57.432483 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0\": container with ID starting with f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0 not found: ID does not exist" containerID="f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0" Oct 31 16:05:57.432567 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.432513 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0"} err="failed to get container status \"f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0\": rpc error: code = NotFound desc = could not find container \"f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0\": container with ID starting with f72fc1e649a68c0a195ee058dec8aebeb1794b7c5b2f1cef254bbe266286b9d0 not found: ID does not exist" Oct 31 16:05:57.580310 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.580284 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f"] Oct 31 16:05:57.629523 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:57.629492 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-190693-openshift-default-797d798599-ht48f"] Oct 31 16:05:58.175517 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.175489 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:58.175881 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.175857 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:58.176809 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.176789 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:05:58.585307 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.585273 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm"] Oct 31 16:05:58.585668 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.585504 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" podUID="db46a452-d867-4b8f-a021-b626034df3a3" containerName="maas-api" containerID="cri-o://90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67" gracePeriod=30 Oct 31 16:05:58.864055 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.864029 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 16:05:58.896116 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.896085 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" path="/var/lib/kubelet/pods/6207a0ed-36ad-4ef4-aa6a-074f0bc1b300/volumes" Oct 31 16:05:58.938994 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.938965 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db46a452-d867-4b8f-a021-b626034df3a3-kube-api-access-6jgsc\" (UniqueName: \"kubernetes.io/projected/db46a452-d867-4b8f-a021-b626034df3a3-kube-api-access-6jgsc\") pod \"db46a452-d867-4b8f-a021-b626034df3a3\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " Oct 31 16:05:58.939188 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.939102 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-metrics-certs\") pod \"db46a452-d867-4b8f-a021-b626034df3a3\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " Oct 31 16:05:58.939188 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.939130 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls\") pod \"db46a452-d867-4b8f-a021-b626034df3a3\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " Oct 31 16:05:58.939188 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.939169 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db46a452-d867-4b8f-a021-b626034df3a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db46a452-d867-4b8f-a021-b626034df3a3-openshift-service-ca\") pod \"db46a452-d867-4b8f-a021-b626034df3a3\" (UID: \"db46a452-d867-4b8f-a021-b626034df3a3\") " Oct 31 16:05:58.939970 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.939936 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db46a452-d867-4b8f-a021-b626034df3a3-openshift-service-ca" pod "db46a452-d867-4b8f-a021-b626034df3a3" (UID: "db46a452-d867-4b8f-a021-b626034df3a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:05:58.941735 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.941682 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db46a452-d867-4b8f-a021-b626034df3a3-kube-api-access-6jgsc" pod "db46a452-d867-4b8f-a021-b626034df3a3" (UID: "db46a452-d867-4b8f-a021-b626034df3a3"). InnerVolumeSpecName "kube-api-access-6jgsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:05:58.941832 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.941682 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls" pod "db46a452-d867-4b8f-a021-b626034df3a3" (UID: "db46a452-d867-4b8f-a021-b626034df3a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:58.942235 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:58.942216 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-metrics-certs" pod "db46a452-d867-4b8f-a021-b626034df3a3" (UID: "db46a452-d867-4b8f-a021-b626034df3a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:05:59.040813 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.040726 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:59.040813 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.040758 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/db46a452-d867-4b8f-a021-b626034df3a3-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:59.040813 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.040771 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/db46a452-d867-4b8f-a021-b626034df3a3-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:59.040813 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.040784 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jgsc\" (UniqueName: \"kubernetes.io/projected/db46a452-d867-4b8f-a021-b626034df3a3-kube-api-access-6jgsc\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:05:59.435368 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.435247 4399 generic.go:356] "Generic (PLEG): container finished" podID="db46a452-d867-4b8f-a021-b626034df3a3" containerID="90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67" exitCode=0 Oct 31 16:05:59.435692 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.435667 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" Oct 31 16:05:59.436118 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.436094 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" event={"ID":"db46a452-d867-4b8f-a021-b626034df3a3","Type":"ContainerDied","Data":"90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67"} Oct 31 16:05:59.436222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.436178 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm" event={"ID":"db46a452-d867-4b8f-a021-b626034df3a3","Type":"ContainerDied","Data":"b8656e8d69d157a285ca344b37dd0c5a7cb3ac944ff012b67d3f445aee82f47e"} Oct 31 16:05:59.436222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.436209 4399 scope.go:122] "RemoveContainer" containerID="90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67" Oct 31 16:05:59.448432 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.448406 4399 scope.go:122] "RemoveContainer" containerID="90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67" Oct 31 16:05:59.448728 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:05:59.448681 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67\": container with ID starting with 90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67 not found: ID does not exist" containerID="90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67" Oct 31 16:05:59.448797 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.448726 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67"} err="failed to get container status \"90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67\": rpc error: code = NotFound desc = could not find container \"90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67\": container with ID starting with 90ebea1ee08b3c74df80f9df8cd4dceffa4c894321c176b3516f568d62069d67 not found: ID does not exist" Oct 31 16:05:59.601998 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.601960 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm"] Oct 31 16:05:59.650746 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:05:59.650683 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-aa864d-5c84d7b77-sq7jm"] Oct 31 16:06:00.895950 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:00.895918 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db46a452-d867-4b8f-a021-b626034df3a3" path="/var/lib/kubelet/pods/db46a452-d867-4b8f-a021-b626034df3a3/volumes" Oct 31 16:06:25.386667 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:25.386631 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn"] Oct 31 16:06:25.518768 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:25.518734 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs"] Oct 31 16:06:26.207087 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:06:26.207052 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:06:30.387244 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:30.387186 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" podUID="7ee41d29-f36e-4c04-93dd-c5b96ad49396" containerName="payload-processing" containerID="cri-o://814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13" gracePeriod=25 Oct 31 16:06:30.519380 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:30.519328 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" podUID="acfbe731-fa58-4f55-954a-daa795b4ec31" containerName="payload-pre-processing" containerID="cri-o://f5045ce07a2e02e32792fb7bb111d8a28504f26208be13e53a7c8ae0847d6c08" gracePeriod=25 Oct 31 16:06:33.277578 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:33.277535 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" podUID="acfbe731-fa58-4f55-954a-daa795b4ec31" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:06:33.277578 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:33.277571 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" podUID="7ee41d29-f36e-4c04-93dd-c5b96ad49396" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:06:43.277537 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:43.277498 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" podUID="7ee41d29-f36e-4c04-93dd-c5b96ad49396" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:06:43.277537 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:43.277518 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" podUID="acfbe731-fa58-4f55-954a-daa795b4ec31" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 16:06:50.647084 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.647029 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 16:06:50.695936 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.695606 4399 generic.go:356] "Generic (PLEG): container finished" podID="acfbe731-fa58-4f55-954a-daa795b4ec31" containerID="f5045ce07a2e02e32792fb7bb111d8a28504f26208be13e53a7c8ae0847d6c08" exitCode=0 Oct 31 16:06:50.695936 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.695853 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" event={"ID":"acfbe731-fa58-4f55-954a-daa795b4ec31","Type":"ContainerDied","Data":"f5045ce07a2e02e32792fb7bb111d8a28504f26208be13e53a7c8ae0847d6c08"} Oct 31 16:06:50.698141 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.698111 4399 generic.go:356] "Generic (PLEG): container finished" podID="7ee41d29-f36e-4c04-93dd-c5b96ad49396" containerID="814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13" exitCode=0 Oct 31 16:06:50.698283 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.698176 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" Oct 31 16:06:50.698283 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.698182 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" event={"ID":"7ee41d29-f36e-4c04-93dd-c5b96ad49396","Type":"ContainerDied","Data":"814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13"} Oct 31 16:06:50.698917 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.698299 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn" event={"ID":"7ee41d29-f36e-4c04-93dd-c5b96ad49396","Type":"ContainerDied","Data":"b572de563fc88ee9b1a8d3bdd8e2a52a69d583d551aeb705e091c9640df44cb4"} Oct 31 16:06:50.698917 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.698333 4399 scope.go:122] "RemoveContainer" containerID="814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13" Oct 31 16:06:50.710820 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.710795 4399 scope.go:122] "RemoveContainer" containerID="814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13" Oct 31 16:06:50.711164 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:06:50.711137 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13\": container with ID starting with 814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13 not found: ID does not exist" containerID="814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13" Oct 31 16:06:50.711263 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.711175 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13"} err="failed to get container status \"814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13\": rpc error: code = NotFound desc = could not find container \"814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13\": container with ID starting with 814a1a6d649d3bfe0cfa15066505b5f5f6af28375fb693f196822d03108eac13 not found: ID does not exist" Oct 31 16:06:50.763887 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.763865 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 16:06:50.773291 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.773268 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee41d29-f36e-4c04-93dd-c5b96ad49396-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee41d29-f36e-4c04-93dd-c5b96ad49396-plugins-config-volume\") pod \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\" (UID: \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\") " Oct 31 16:06:50.773419 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.773335 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee41d29-f36e-4c04-93dd-c5b96ad49396-kube-api-access-jrx2s\" (UniqueName: \"kubernetes.io/projected/7ee41d29-f36e-4c04-93dd-c5b96ad49396-kube-api-access-jrx2s\") pod \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\" (UID: \"7ee41d29-f36e-4c04-93dd-c5b96ad49396\") " Oct 31 16:06:50.773608 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.773586 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee41d29-f36e-4c04-93dd-c5b96ad49396-plugins-config-volume" pod "7ee41d29-f36e-4c04-93dd-c5b96ad49396" (UID: "7ee41d29-f36e-4c04-93dd-c5b96ad49396"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:06:50.773751 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.773737 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ee41d29-f36e-4c04-93dd-c5b96ad49396-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:50.775682 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.775656 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee41d29-f36e-4c04-93dd-c5b96ad49396-kube-api-access-jrx2s" pod "7ee41d29-f36e-4c04-93dd-c5b96ad49396" (UID: "7ee41d29-f36e-4c04-93dd-c5b96ad49396"). InnerVolumeSpecName "kube-api-access-jrx2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:50.879422 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.875161 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acfbe731-fa58-4f55-954a-daa795b4ec31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfbe731-fa58-4f55-954a-daa795b4ec31-plugins-config-volume\") pod \"acfbe731-fa58-4f55-954a-daa795b4ec31\" (UID: \"acfbe731-fa58-4f55-954a-daa795b4ec31\") " Oct 31 16:06:50.879422 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.875239 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acfbe731-fa58-4f55-954a-daa795b4ec31-kube-api-access-2mzm2\" (UniqueName: \"kubernetes.io/projected/acfbe731-fa58-4f55-954a-daa795b4ec31-kube-api-access-2mzm2\") pod \"acfbe731-fa58-4f55-954a-daa795b4ec31\" (UID: \"acfbe731-fa58-4f55-954a-daa795b4ec31\") " Oct 31 16:06:50.879422 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.875757 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrx2s\" (UniqueName: \"kubernetes.io/projected/7ee41d29-f36e-4c04-93dd-c5b96ad49396-kube-api-access-jrx2s\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:50.879422 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.877603 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acfbe731-fa58-4f55-954a-daa795b4ec31-plugins-config-volume" pod "acfbe731-fa58-4f55-954a-daa795b4ec31" (UID: "acfbe731-fa58-4f55-954a-daa795b4ec31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:06:50.880252 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.880227 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acfbe731-fa58-4f55-954a-daa795b4ec31-kube-api-access-2mzm2" pod "acfbe731-fa58-4f55-954a-daa795b4ec31" (UID: "acfbe731-fa58-4f55-954a-daa795b4ec31"). InnerVolumeSpecName "kube-api-access-2mzm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:06:50.976610 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.976582 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acfbe731-fa58-4f55-954a-daa795b4ec31-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:50.976610 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:50.976606 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mzm2\" (UniqueName: \"kubernetes.io/projected/acfbe731-fa58-4f55-954a-daa795b4ec31-kube-api-access-2mzm2\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:06:51.113681 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:51.113641 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn"] Oct 31 16:06:51.117472 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:51.117446 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2348-6bcc5d967f-kq8bn"] Oct 31 16:06:51.704048 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:51.704018 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" Oct 31 16:06:51.704048 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:51.704028 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs" event={"ID":"acfbe731-fa58-4f55-954a-daa795b4ec31","Type":"ContainerDied","Data":"c7a793715045b78502607efba93d5577e052a7fd066095467c93b9caedb6bc4f"} Oct 31 16:06:51.704745 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:51.704096 4399 scope.go:122] "RemoveContainer" containerID="f5045ce07a2e02e32792fb7bb111d8a28504f26208be13e53a7c8ae0847d6c08" Oct 31 16:06:51.867097 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:51.867067 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs"] Oct 31 16:06:51.868882 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:51.868862 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2348-784ccf56bc-q4lfs"] Oct 31 16:06:52.895179 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:52.895144 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee41d29-f36e-4c04-93dd-c5b96ad49396" path="/var/lib/kubelet/pods/7ee41d29-f36e-4c04-93dd-c5b96ad49396/volumes" Oct 31 16:06:52.895689 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:06:52.895675 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acfbe731-fa58-4f55-954a-daa795b4ec31" path="/var/lib/kubelet/pods/acfbe731-fa58-4f55-954a-daa795b4ec31/volumes" Oct 31 16:06:56.206463 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:06:56.206431 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:07:21.779732 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:21.779682 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff"] Oct 31 16:07:21.780272 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:21.779896 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" containerID="cri-o://2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a" gracePeriod=30 Oct 31 16:07:26.209093 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:07:26.209055 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:07:26.367101 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:26.367061 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-gvqff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:07:26.367270 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:26.367130 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:07:27.048222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.048190 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 16:07:27.118990 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.118916 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-data\") pod \"148ec614-34ae-494b-8652-d1865243f5b5\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " Oct 31 16:07:27.118990 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.118975 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-istio-token\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-istio-token\") pod \"148ec614-34ae-494b-8652-d1865243f5b5\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " Oct 31 16:07:27.119222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119028 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-kube-api-access-qw54c\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-kube-api-access-qw54c\") pod \"148ec614-34ae-494b-8652-d1865243f5b5\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " Oct 31 16:07:27.119222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119094 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-socket\") pod \"148ec614-34ae-494b-8652-d1865243f5b5\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " Oct 31 16:07:27.119222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119152 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-certs\") pod \"148ec614-34ae-494b-8652-d1865243f5b5\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " Oct 31 16:07:27.119222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119156 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-data" pod "148ec614-34ae-494b-8652-d1865243f5b5" (UID: "148ec614-34ae-494b-8652-d1865243f5b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:27.119222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119202 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/148ec614-34ae-494b-8652-d1865243f5b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ec614-34ae-494b-8652-d1865243f5b5-istio-podinfo\") pod \"148ec614-34ae-494b-8652-d1865243f5b5\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " Oct 31 16:07:27.119522 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119238 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-credential-socket\") pod \"148ec614-34ae-494b-8652-d1865243f5b5\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " Oct 31 16:07:27.119522 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119298 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-envoy\") pod \"148ec614-34ae-494b-8652-d1865243f5b5\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " Oct 31 16:07:27.119522 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119344 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/148ec614-34ae-494b-8652-d1865243f5b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ec614-34ae-494b-8652-d1865243f5b5-istiod-ca-cert\") pod \"148ec614-34ae-494b-8652-d1865243f5b5\" (UID: \"148ec614-34ae-494b-8652-d1865243f5b5\") " Oct 31 16:07:27.119522 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119363 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-socket" pod "148ec614-34ae-494b-8652-d1865243f5b5" (UID: "148ec614-34ae-494b-8652-d1865243f5b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:27.119522 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119475 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-certs" pod "148ec614-34ae-494b-8652-d1865243f5b5" (UID: "148ec614-34ae-494b-8652-d1865243f5b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:27.120013 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119575 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-credential-socket" pod "148ec614-34ae-494b-8652-d1865243f5b5" (UID: "148ec614-34ae-494b-8652-d1865243f5b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:27.120013 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119796 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/148ec614-34ae-494b-8652-d1865243f5b5-istiod-ca-cert" pod "148ec614-34ae-494b-8652-d1865243f5b5" (UID: "148ec614-34ae-494b-8652-d1865243f5b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:07:27.120013 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119919 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:27.120013 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119936 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:27.120013 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119951 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:27.120013 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119965 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/148ec614-34ae-494b-8652-d1865243f5b5-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:27.120013 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.119979 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:27.121514 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.121487 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-kube-api-access-qw54c" pod "148ec614-34ae-494b-8652-d1865243f5b5" (UID: "148ec614-34ae-494b-8652-d1865243f5b5"). InnerVolumeSpecName "kube-api-access-qw54c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:07:27.121647 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.121587 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/148ec614-34ae-494b-8652-d1865243f5b5-istio-podinfo" pod "148ec614-34ae-494b-8652-d1865243f5b5" (UID: "148ec614-34ae-494b-8652-d1865243f5b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:07:27.121830 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.121800 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-istio-token" pod "148ec614-34ae-494b-8652-d1865243f5b5" (UID: "148ec614-34ae-494b-8652-d1865243f5b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:07:27.121919 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.121904 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-envoy" pod "148ec614-34ae-494b-8652-d1865243f5b5" (UID: "148ec614-34ae-494b-8652-d1865243f5b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:07:27.221059 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.221025 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/148ec614-34ae-494b-8652-d1865243f5b5-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:27.221059 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.221059 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:27.221453 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.221075 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw54c\" (UniqueName: \"kubernetes.io/projected/148ec614-34ae-494b-8652-d1865243f5b5-kube-api-access-qw54c\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:27.221453 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.221090 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/148ec614-34ae-494b-8652-d1865243f5b5-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:07:27.880507 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.880474 4399 generic.go:356] "Generic (PLEG): container finished" podID="148ec614-34ae-494b-8652-d1865243f5b5" containerID="2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a" exitCode=0 Oct 31 16:07:27.880722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.880524 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" Oct 31 16:07:27.880722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.880571 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" event={"ID":"148ec614-34ae-494b-8652-d1865243f5b5","Type":"ContainerDied","Data":"2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a"} Oct 31 16:07:27.880722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.880626 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff" event={"ID":"148ec614-34ae-494b-8652-d1865243f5b5","Type":"ContainerDied","Data":"f7bb61d0e8f4640378543cf61a8f36c593095f04618f975cbb9c6bd73cf0b1f4"} Oct 31 16:07:27.880722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.880650 4399 scope.go:122] "RemoveContainer" containerID="2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a" Oct 31 16:07:27.890531 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.890517 4399 scope.go:122] "RemoveContainer" containerID="2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a" Oct 31 16:07:27.890892 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:07:27.890870 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a\": container with ID starting with 2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a not found: ID does not exist" containerID="2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a" Oct 31 16:07:27.890976 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:27.890897 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a"} err="failed to get container status \"2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a\": rpc error: code = NotFound desc = could not find container \"2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a\": container with ID starting with 2fb66b82d249fbb22df2de32e6f070740361611eab7a6c55fe41c1935774f44a not found: ID does not exist" Oct 31 16:07:28.095302 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:28.095268 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff"] Oct 31 16:07:28.144961 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:28.144881 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-gvqff"] Oct 31 16:07:28.895613 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:07:28.895579 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="148ec614-34ae-494b-8652-d1865243f5b5" path="/var/lib/kubelet/pods/148ec614-34ae-494b-8652-d1865243f5b5/volumes" Oct 31 16:07:56.206264 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:07:56.206227 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:08:26.208554 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:08:26.208515 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:08:56.205955 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:08:56.205919 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:09:10.571916 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:10.571825 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22"] Oct 31 16:09:10.572343 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:10.572101 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" containerID="cri-o://284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6" gracePeriod=30 Oct 31 16:09:13.178073 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:13.178035 4399 patch_prober.go:29] interesting pod/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:09:13.178464 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:13.178111 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:09:15.830585 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.830562 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:09:15.962505 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.962478 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-credential-socket\") pod \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " Oct 31 16:09:15.962679 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.962521 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istiod-ca-cert\") pod \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " Oct 31 16:09:15.962679 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.962561 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-socket\") pod \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " Oct 31 16:09:15.962679 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.962597 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-kube-api-access-jk885\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-kube-api-access-jk885\") pod \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " Oct 31 16:09:15.962679 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.962628 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-podinfo\") pod \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " Oct 31 16:09:15.962679 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.962662 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-envoy\") pod \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " Oct 31 16:09:15.962945 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.962746 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-data\") pod \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " Oct 31 16:09:15.962945 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.962795 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-certs\") pod \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " Oct 31 16:09:15.962945 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.962831 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-token\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-token\") pod \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\" (UID: \"75d96ab8-723c-4e78-9f73-83cbcf01c7b3\") " Oct 31 16:09:15.963126 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.963002 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-socket" pod "75d96ab8-723c-4e78-9f73-83cbcf01c7b3" (UID: "75d96ab8-723c-4e78-9f73-83cbcf01c7b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.963471 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.963270 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:15.963471 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.963351 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-data" pod "75d96ab8-723c-4e78-9f73-83cbcf01c7b3" (UID: "75d96ab8-723c-4e78-9f73-83cbcf01c7b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.963661 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.963484 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-certs" pod "75d96ab8-723c-4e78-9f73-83cbcf01c7b3" (UID: "75d96ab8-723c-4e78-9f73-83cbcf01c7b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.963988 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.963962 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-credential-socket" pod "75d96ab8-723c-4e78-9f73-83cbcf01c7b3" (UID: "75d96ab8-723c-4e78-9f73-83cbcf01c7b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.964526 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.964497 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istiod-ca-cert" pod "75d96ab8-723c-4e78-9f73-83cbcf01c7b3" (UID: "75d96ab8-723c-4e78-9f73-83cbcf01c7b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:09:15.965749 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.965728 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-podinfo" pod "75d96ab8-723c-4e78-9f73-83cbcf01c7b3" (UID: "75d96ab8-723c-4e78-9f73-83cbcf01c7b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:09:15.965837 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.965761 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-token" pod "75d96ab8-723c-4e78-9f73-83cbcf01c7b3" (UID: "75d96ab8-723c-4e78-9f73-83cbcf01c7b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:15.965914 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.965895 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-envoy" pod "75d96ab8-723c-4e78-9f73-83cbcf01c7b3" (UID: "75d96ab8-723c-4e78-9f73-83cbcf01c7b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:15.966598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:15.966579 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-kube-api-access-jk885" pod "75d96ab8-723c-4e78-9f73-83cbcf01c7b3" (UID: "75d96ab8-723c-4e78-9f73-83cbcf01c7b3"). InnerVolumeSpecName "kube-api-access-jk885". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:16.063982 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.063950 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.063982 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.063976 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.063982 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.063987 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk885\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-kube-api-access-jk885\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.064260 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.064011 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.064260 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.064025 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.064260 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.064037 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.064260 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.064047 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.064260 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.064055 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75d96ab8-723c-4e78-9f73-83cbcf01c7b3-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:16.417857 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.417768 4399 generic.go:356] "Generic (PLEG): container finished" podID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerID="284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6" exitCode=0 Oct 31 16:09:16.417857 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.417816 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" Oct 31 16:09:16.418062 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.417852 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" event={"ID":"75d96ab8-723c-4e78-9f73-83cbcf01c7b3","Type":"ContainerDied","Data":"284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6"} Oct 31 16:09:16.418062 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.417911 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22" event={"ID":"75d96ab8-723c-4e78-9f73-83cbcf01c7b3","Type":"ContainerDied","Data":"64992956862b5af7c1f3186725dbd947a898d7d122f6a26ca25b753f9b688554"} Oct 31 16:09:16.418062 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.417943 4399 scope.go:122] "RemoveContainer" containerID="284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6" Oct 31 16:09:16.427813 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.427796 4399 scope.go:122] "RemoveContainer" containerID="284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6" Oct 31 16:09:16.428102 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:09:16.428083 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6\": container with ID starting with 284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6 not found: ID does not exist" containerID="284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6" Oct 31 16:09:16.428183 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.428113 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6"} err="failed to get container status \"284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6\": rpc error: code = NotFound desc = could not find container \"284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6\": container with ID starting with 284f0d384b687a280bd5573b81e145fafe4899a8f71323242fda5d23370213c6 not found: ID does not exist" Oct 31 16:09:16.656722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.656670 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22"] Oct 31 16:09:16.769293 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.769265 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-31dff8-openshift-default-66c57c7579-dbl22"] Oct 31 16:09:16.896907 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:16.896873 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" path="/var/lib/kubelet/pods/75d96ab8-723c-4e78-9f73-83cbcf01c7b3/volumes" Oct 31 16:09:26.205740 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:09:26.205678 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:09:30.386103 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:30.386066 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9"] Oct 31 16:09:45.406550 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.406480 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" containerID="cri-o://9445dd8b7f12e145b48caca0d284f26bcd767ddc19a6555589f2c6584e688a0b" gracePeriod=45 Oct 31 16:09:45.429411 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.429378 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/ready\": dial tcp 10.133.0.58:8000: connect: connection refused" Oct 31 16:09:45.580849 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.580818 4399 generic.go:356] "Generic (PLEG): container finished" podID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerID="9445dd8b7f12e145b48caca0d284f26bcd767ddc19a6555589f2c6584e688a0b" exitCode=0 Oct 31 16:09:45.581091 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.580874 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" event={"ID":"79e5d115-2489-4b78-a250-6f3ef934b12b","Type":"ContainerDied","Data":"9445dd8b7f12e145b48caca0d284f26bcd767ddc19a6555589f2c6584e688a0b"} Oct 31 16:09:45.660436 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.660360 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 16:09:45.737383 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737349 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-home\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-home\") pod \"79e5d115-2489-4b78-a250-6f3ef934b12b\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " Oct 31 16:09:45.737564 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737404 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-dshm\") pod \"79e5d115-2489-4b78-a250-6f3ef934b12b\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " Oct 31 16:09:45.737564 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737435 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-model-cache\") pod \"79e5d115-2489-4b78-a250-6f3ef934b12b\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " Oct 31 16:09:45.737564 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737501 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e5d115-2489-4b78-a250-6f3ef934b12b-tls-certs\" (UniqueName: \"kubernetes.io/secret/79e5d115-2489-4b78-a250-6f3ef934b12b-tls-certs\") pod \"79e5d115-2489-4b78-a250-6f3ef934b12b\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " Oct 31 16:09:45.737564 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737546 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-tmp-dir\") pod \"79e5d115-2489-4b78-a250-6f3ef934b12b\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " Oct 31 16:09:45.737835 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737573 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e5d115-2489-4b78-a250-6f3ef934b12b-kube-api-access-plsf5\" (UniqueName: \"kubernetes.io/projected/79e5d115-2489-4b78-a250-6f3ef934b12b-kube-api-access-plsf5\") pod \"79e5d115-2489-4b78-a250-6f3ef934b12b\" (UID: \"79e5d115-2489-4b78-a250-6f3ef934b12b\") " Oct 31 16:09:45.737835 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737722 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-home" pod "79e5d115-2489-4b78-a250-6f3ef934b12b" (UID: "79e5d115-2489-4b78-a250-6f3ef934b12b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.737835 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737783 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-model-cache" pod "79e5d115-2489-4b78-a250-6f3ef934b12b" (UID: "79e5d115-2489-4b78-a250-6f3ef934b12b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.738005 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737878 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-tmp-dir" pod "79e5d115-2489-4b78-a250-6f3ef934b12b" (UID: "79e5d115-2489-4b78-a250-6f3ef934b12b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.738005 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737963 4399 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-home\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.738005 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.737978 4399 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-model-cache\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.740065 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.740039 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e5d115-2489-4b78-a250-6f3ef934b12b-tls-certs" pod "79e5d115-2489-4b78-a250-6f3ef934b12b" (UID: "79e5d115-2489-4b78-a250-6f3ef934b12b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:09:45.740181 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.740068 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e5d115-2489-4b78-a250-6f3ef934b12b-kube-api-access-plsf5" pod "79e5d115-2489-4b78-a250-6f3ef934b12b" (UID: "79e5d115-2489-4b78-a250-6f3ef934b12b"). InnerVolumeSpecName "kube-api-access-plsf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:45.740181 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.740134 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-dshm" pod "79e5d115-2489-4b78-a250-6f3ef934b12b" (UID: "79e5d115-2489-4b78-a250-6f3ef934b12b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:09:45.839651 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.839606 4399 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-tmp-dir\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.839651 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.839652 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plsf5\" (UniqueName: \"kubernetes.io/projected/79e5d115-2489-4b78-a250-6f3ef934b12b-kube-api-access-plsf5\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.839935 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.839671 4399 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/79e5d115-2489-4b78-a250-6f3ef934b12b-dshm\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:45.839935 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:45.839687 4399 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/79e5d115-2489-4b78-a250-6f3ef934b12b-tls-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:46.586843 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:46.586807 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" Oct 31 16:09:46.587297 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:46.586834 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9" event={"ID":"79e5d115-2489-4b78-a250-6f3ef934b12b","Type":"ContainerDied","Data":"5285a65f85183acf4fd3c55938e69286fc111849826c9ecdfc3e6b0b031072ae"} Oct 31 16:09:46.587297 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:46.586890 4399 scope.go:122] "RemoveContainer" containerID="9445dd8b7f12e145b48caca0d284f26bcd767ddc19a6555589f2c6584e688a0b" Oct 31 16:09:46.769487 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:46.769450 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9"] Oct 31 16:09:46.828161 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:46.828133 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-56bc96/premium-w0-56bc96-kserve-6577499bc9-45dh9"] Oct 31 16:09:46.895605 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:46.895535 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" path="/var/lib/kubelet/pods/79e5d115-2489-4b78-a250-6f3ef934b12b/volumes" Oct 31 16:09:52.251809 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.251771 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd"] Oct 31 16:09:52.252296 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.251994 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" podUID="60c8879b-fee1-48f8-8b51-3901a4dea925" containerName="maas-api" containerID="cri-o://a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864" gracePeriod=30 Oct 31 16:09:52.500648 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.500624 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 16:09:52.601842 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.601780 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-metrics-certs\") pod \"60c8879b-fee1-48f8-8b51-3901a4dea925\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " Oct 31 16:09:52.601842 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.601816 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls\") pod \"60c8879b-fee1-48f8-8b51-3901a4dea925\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " Oct 31 16:09:52.602051 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.601851 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c8879b-fee1-48f8-8b51-3901a4dea925-kube-api-access-88dns\" (UniqueName: \"kubernetes.io/projected/60c8879b-fee1-48f8-8b51-3901a4dea925-kube-api-access-88dns\") pod \"60c8879b-fee1-48f8-8b51-3901a4dea925\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " Oct 31 16:09:52.602051 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.601881 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c8879b-fee1-48f8-8b51-3901a4dea925-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c8879b-fee1-48f8-8b51-3901a4dea925-openshift-service-ca\") pod \"60c8879b-fee1-48f8-8b51-3901a4dea925\" (UID: \"60c8879b-fee1-48f8-8b51-3901a4dea925\") " Oct 31 16:09:52.602331 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.602297 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c8879b-fee1-48f8-8b51-3901a4dea925-openshift-service-ca" pod "60c8879b-fee1-48f8-8b51-3901a4dea925" (UID: "60c8879b-fee1-48f8-8b51-3901a4dea925"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:09:52.604116 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.604096 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-metrics-certs" pod "60c8879b-fee1-48f8-8b51-3901a4dea925" (UID: "60c8879b-fee1-48f8-8b51-3901a4dea925"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:09:52.604207 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.604115 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls" pod "60c8879b-fee1-48f8-8b51-3901a4dea925" (UID: "60c8879b-fee1-48f8-8b51-3901a4dea925"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:09:52.604207 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.604130 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c8879b-fee1-48f8-8b51-3901a4dea925-kube-api-access-88dns" pod "60c8879b-fee1-48f8-8b51-3901a4dea925" (UID: "60c8879b-fee1-48f8-8b51-3901a4dea925"). InnerVolumeSpecName "kube-api-access-88dns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:09:52.615971 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.615948 4399 generic.go:356] "Generic (PLEG): container finished" podID="60c8879b-fee1-48f8-8b51-3901a4dea925" containerID="a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864" exitCode=0 Oct 31 16:09:52.616067 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.615991 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" event={"ID":"60c8879b-fee1-48f8-8b51-3901a4dea925","Type":"ContainerDied","Data":"a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864"} Oct 31 16:09:52.616067 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.616004 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" Oct 31 16:09:52.616067 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.616051 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd" event={"ID":"60c8879b-fee1-48f8-8b51-3901a4dea925","Type":"ContainerDied","Data":"bdb2300cb07187fc18425fb18dc8c1c086e0451fc816675e792d0dcb9c18d27a"} Oct 31 16:09:52.616187 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.616082 4399 scope.go:122] "RemoveContainer" containerID="a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864" Oct 31 16:09:52.628313 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.628071 4399 scope.go:122] "RemoveContainer" containerID="a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864" Oct 31 16:09:52.628468 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:09:52.628430 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864\": container with ID starting with a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864 not found: ID does not exist" containerID="a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864" Oct 31 16:09:52.628524 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.628462 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864"} err="failed to get container status \"a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864\": rpc error: code = NotFound desc = could not find container \"a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864\": container with ID starting with a5a3218a388e463528886940f63e32648536e8552b225892fcf9cab62c515864 not found: ID does not exist" Oct 31 16:09:52.703328 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.703295 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88dns\" (UniqueName: \"kubernetes.io/projected/60c8879b-fee1-48f8-8b51-3901a4dea925-kube-api-access-88dns\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:52.703328 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.703321 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/60c8879b-fee1-48f8-8b51-3901a4dea925-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:52.703328 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.703332 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:52.703568 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.703340 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/60c8879b-fee1-48f8-8b51-3901a4dea925-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:09:52.787001 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.786972 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd"] Oct 31 16:09:52.843557 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.843528 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-56bc96-568b98b895-299hd"] Oct 31 16:09:52.895435 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:52.895372 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c8879b-fee1-48f8-8b51-3901a4dea925" path="/var/lib/kubelet/pods/60c8879b-fee1-48f8-8b51-3901a4dea925/volumes" Oct 31 16:09:56.175918 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.175888 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh"] Oct 31 16:09:56.176342 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176329 4399 cpu_manager.go:414] "containerMap: removing container" podUID="7ee41d29-f36e-4c04-93dd-c5b96ad49396" containerName="payload-processing" Oct 31 16:09:56.176391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176342 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee41d29-f36e-4c04-93dd-c5b96ad49396" containerName="payload-processing" Oct 31 16:09:56.176391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176355 4399 cpu_manager.go:414] "containerMap: removing container" podUID="60c8879b-fee1-48f8-8b51-3901a4dea925" containerName="maas-api" Oct 31 16:09:56.176391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176360 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c8879b-fee1-48f8-8b51-3901a4dea925" containerName="maas-api" Oct 31 16:09:56.176391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176371 4399 cpu_manager.go:414] "containerMap: removing container" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" Oct 31 16:09:56.176391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176376 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" Oct 31 16:09:56.176391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176382 4399 cpu_manager.go:414] "containerMap: removing container" podUID="db46a452-d867-4b8f-a021-b626034df3a3" containerName="maas-api" Oct 31 16:09:56.176391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176386 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db46a452-d867-4b8f-a021-b626034df3a3" containerName="maas-api" Oct 31 16:09:56.176391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176394 4399 cpu_manager.go:414] "containerMap: removing container" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176398 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176420 4399 cpu_manager.go:414] "containerMap: removing container" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176426 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176435 4399 cpu_manager.go:414] "containerMap: removing container" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176439 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176444 4399 cpu_manager.go:414] "containerMap: removing container" podUID="acfbe731-fa58-4f55-954a-daa795b4ec31" containerName="payload-pre-processing" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176448 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acfbe731-fa58-4f55-954a-daa795b4ec31" containerName="payload-pre-processing" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176460 4399 cpu_manager.go:414] "containerMap: removing container" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176464 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176520 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176530 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee41d29-f36e-4c04-93dd-c5b96ad49396" containerName="payload-processing" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176537 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c8879b-fee1-48f8-8b51-3901a4dea925" containerName="maas-api" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176545 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="db46a452-d867-4b8f-a021-b626034df3a3" containerName="maas-api" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176552 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="148ec614-34ae-494b-8652-d1865243f5b5" containerName="istio-proxy" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176560 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d96ab8-723c-4e78-9f73-83cbcf01c7b3" containerName="istio-proxy" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176565 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e5d115-2489-4b78-a250-6f3ef934b12b" containerName="main" Oct 31 16:09:56.176650 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.176570 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="acfbe731-fa58-4f55-954a-daa795b4ec31" containerName="payload-pre-processing" Oct 31 16:09:56.182145 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.182123 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.206683 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:09:56.206648 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:09:56.233437 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.233405 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328607f0-636e-4f5b-8e94-c2132ad25fbd-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.233630 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.233449 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hrq\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-kube-api-access-m6hrq\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.233630 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.233470 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.233815 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.233642 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.233815 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.233681 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.233936 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.233822 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.233936 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.233865 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.233936 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.233894 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.234039 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.233987 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.240557 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.240530 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-56bc96-openshift-default-dockercfg-5qxbd\"" Oct 31 16:09:56.299399 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.299366 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh"] Oct 31 16:09:56.335545 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.335505 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328607f0-636e-4f5b-8e94-c2132ad25fbd-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.335798 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.335564 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hrq\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-kube-api-access-m6hrq\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.335798 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.335729 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.335965 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.335824 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.335965 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.335854 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.335965 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.335898 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.335965 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.335934 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.335965 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.335963 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.336232 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.336027 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.336384 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.336361 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-data\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.336487 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.336461 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328607f0-636e-4f5b-8e94-c2132ad25fbd-istiod-ca-cert\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.336597 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.336570 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-credential-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.336811 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.336589 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-certs\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.336911 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.336610 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-socket\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.338817 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.338785 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-envoy\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.339137 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.339121 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-podinfo\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.423363 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.423333 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-token\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.423534 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.423508 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hrq\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-kube-api-access-m6hrq\") pod \"e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.495388 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.495358 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:56.938599 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:09:56.938562 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328607f0_636e_4f5b_8e94_c2132ad25fbd.slice/crio-6ea951c444fb3460c2e8f3a9115daca35b8aa2bc2603aafadff002d7c34e1829 WatchSource:0}: Error finding container 6ea951c444fb3460c2e8f3a9115daca35b8aa2bc2603aafadff002d7c34e1829: Status 404 returned error can't find the container with id 6ea951c444fb3460c2e8f3a9115daca35b8aa2bc2603aafadff002d7c34e1829 Oct 31 16:09:56.940323 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.940300 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh"] Oct 31 16:09:56.940953 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.940919 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:09:56.941060 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.940996 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:09:56.941060 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:56.941040 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:09:57.657724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:57.653264 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" event={"ID":"328607f0-636e-4f5b-8e94-c2132ad25fbd","Type":"ContainerStarted","Data":"43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e"} Oct 31 16:09:57.657724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:57.653332 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" event={"ID":"328607f0-636e-4f5b-8e94-c2132ad25fbd","Type":"ContainerStarted","Data":"6ea951c444fb3460c2e8f3a9115daca35b8aa2bc2603aafadff002d7c34e1829"} Oct 31 16:09:57.907430 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:57.907377 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" podStartSLOduration=1.9073623469999998 podStartE2EDuration="1.907362347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:09:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:09:57.905760063 +0000 UTC m=+5041.117001472" watchObservedRunningTime="2026-10-31 16:09:57.907362347 +0000 UTC m=+5041.118603758" Oct 31 16:09:58.495940 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:58.495792 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:09:58.497622 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:58.497596 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 16:09:58.497760 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:58.497651 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 16:09:59.495850 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:59.495817 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 16:09:59.496269 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:09:59.495869 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 16:10:00.495798 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:00.495759 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 16:10:00.496087 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:00.495829 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 16:10:01.495934 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:01.495902 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 16:10:01.496120 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:01.495954 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 16:10:02.496134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:02.496095 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 16:10:02.496568 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:02.496155 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 16:10:03.496069 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:03.496026 4399 patch_prober.go:29] interesting pod/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 31 16:10:03.496264 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:03.496100 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 31 16:10:05.012812 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:05.012772 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:10:05.013260 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:05.013096 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:10:05.041350 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:05.041329 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:10:10.288978 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:10.288945 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh"] Oct 31 16:10:10.289393 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:10.289171 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" containerID="cri-o://43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e" gracePeriod=30 Oct 31 16:10:11.379567 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:11.379537 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b9bfdbc7b-7jf7k_1404f6c8-54fc-49bc-b514-3e6956e20350/authorino/0.log" Oct 31 16:10:13.827846 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:13.827812 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-s7mjs_338dae66-850f-4bbc-ae9e-da170cf58631/manager/0.log" Oct 31 16:10:14.904652 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:14.904621 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-78f488d89-cv2qm_bd10b679-de31-4629-8256-cf75f52e76c4/manager/0.log" Oct 31 16:10:15.457976 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.457950 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p2f7_e191e1bb-a406-4fa3-b9bf-b27502111962/manager/0.log" Oct 31 16:10:15.546955 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.546930 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:10:15.647636 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647560 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-token\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-token\") pod \"328607f0-636e-4f5b-8e94-c2132ad25fbd\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " Oct 31 16:10:15.647636 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647632 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-certs\") pod \"328607f0-636e-4f5b-8e94-c2132ad25fbd\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " Oct 31 16:10:15.647880 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647685 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-credential-socket\") pod \"328607f0-636e-4f5b-8e94-c2132ad25fbd\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " Oct 31 16:10:15.647880 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647743 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-socket\") pod \"328607f0-636e-4f5b-8e94-c2132ad25fbd\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " Oct 31 16:10:15.647880 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647771 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-envoy\") pod \"328607f0-636e-4f5b-8e94-c2132ad25fbd\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " Oct 31 16:10:15.647880 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647808 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/328607f0-636e-4f5b-8e94-c2132ad25fbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328607f0-636e-4f5b-8e94-c2132ad25fbd-istiod-ca-cert\") pod \"328607f0-636e-4f5b-8e94-c2132ad25fbd\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " Oct 31 16:10:15.647880 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647837 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-kube-api-access-m6hrq\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-kube-api-access-m6hrq\") pod \"328607f0-636e-4f5b-8e94-c2132ad25fbd\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " Oct 31 16:10:15.647880 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647864 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-data\") pod \"328607f0-636e-4f5b-8e94-c2132ad25fbd\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " Oct 31 16:10:15.648212 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647898 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-podinfo\") pod \"328607f0-636e-4f5b-8e94-c2132ad25fbd\" (UID: \"328607f0-636e-4f5b-8e94-c2132ad25fbd\") " Oct 31 16:10:15.648212 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.647981 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-certs" pod "328607f0-636e-4f5b-8e94-c2132ad25fbd" (UID: "328607f0-636e-4f5b-8e94-c2132ad25fbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.648333 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.648311 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.648492 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.648452 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-credential-socket" pod "328607f0-636e-4f5b-8e94-c2132ad25fbd" (UID: "328607f0-636e-4f5b-8e94-c2132ad25fbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.648492 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.648452 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/328607f0-636e-4f5b-8e94-c2132ad25fbd-istiod-ca-cert" pod "328607f0-636e-4f5b-8e94-c2132ad25fbd" (UID: "328607f0-636e-4f5b-8e94-c2132ad25fbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:10:15.648887 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.648530 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-socket" pod "328607f0-636e-4f5b-8e94-c2132ad25fbd" (UID: "328607f0-636e-4f5b-8e94-c2132ad25fbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.648887 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.648617 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-data" pod "328607f0-636e-4f5b-8e94-c2132ad25fbd" (UID: "328607f0-636e-4f5b-8e94-c2132ad25fbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.650204 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.650177 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-envoy" pod "328607f0-636e-4f5b-8e94-c2132ad25fbd" (UID: "328607f0-636e-4f5b-8e94-c2132ad25fbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:10:15.650204 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.650194 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-token" pod "328607f0-636e-4f5b-8e94-c2132ad25fbd" (UID: "328607f0-636e-4f5b-8e94-c2132ad25fbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:10:15.650333 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.650234 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-podinfo" pod "328607f0-636e-4f5b-8e94-c2132ad25fbd" (UID: "328607f0-636e-4f5b-8e94-c2132ad25fbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:10:15.650462 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.650431 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-kube-api-access-m6hrq" pod "328607f0-636e-4f5b-8e94-c2132ad25fbd" (UID: "328607f0-636e-4f5b-8e94-c2132ad25fbd"). InnerVolumeSpecName "kube-api-access-m6hrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:10:15.749740 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.749671 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.749961 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.749770 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.749961 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.749788 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/328607f0-636e-4f5b-8e94-c2132ad25fbd-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.749961 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.749820 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hrq\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-kube-api-access-m6hrq\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.749961 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.749839 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.749961 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.749853 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.749961 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.749867 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/328607f0-636e-4f5b-8e94-c2132ad25fbd-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.749961 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.749881 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/328607f0-636e-4f5b-8e94-c2132ad25fbd-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:10:15.760516 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.760479 4399 generic.go:356] "Generic (PLEG): container finished" podID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerID="43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e" exitCode=0 Oct 31 16:10:15.760680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.760527 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" Oct 31 16:10:15.760680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.760556 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" event={"ID":"328607f0-636e-4f5b-8e94-c2132ad25fbd","Type":"ContainerDied","Data":"43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e"} Oct 31 16:10:15.760680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.760620 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh" event={"ID":"328607f0-636e-4f5b-8e94-c2132ad25fbd","Type":"ContainerDied","Data":"6ea951c444fb3460c2e8f3a9115daca35b8aa2bc2603aafadff002d7c34e1829"} Oct 31 16:10:15.760680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.760650 4399 scope.go:122] "RemoveContainer" containerID="43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e" Oct 31 16:10:15.770869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.770847 4399 scope.go:122] "RemoveContainer" containerID="43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e" Oct 31 16:10:15.771178 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:10:15.771149 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e\": container with ID starting with 43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e not found: ID does not exist" containerID="43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e" Oct 31 16:10:15.771278 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.771191 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e"} err="failed to get container status \"43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e\": rpc error: code = NotFound desc = could not find container \"43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e\": container with ID starting with 43f6ca742d432c10ac210ee450cafc8eea6e2f95ab3751011ab13b3dc639232e not found: ID does not exist" Oct 31 16:10:15.954362 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:15.954331 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh"] Oct 31 16:10:16.010859 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:16.010826 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-56bc96-openshift-default-9d947bf5f-6txrh"] Oct 31 16:10:16.896074 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:16.896045 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" path="/var/lib/kubelet/pods/328607f0-636e-4f5b-8e94-c2132ad25fbd/volumes" Oct 31 16:10:17.713761 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:17.713730 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knls7_698a358f-ef89-4279-a36c-dd211218127e/server/0.log" Oct 31 16:10:18.344611 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:10:18.344583 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-28xqm_f35f6cb1-6438-4d87-a0ac-078e5960373e/proxy/0.log" Oct 31 16:10:26.206612 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:10:26.206575 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:10:56.206042 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:10:56.206008 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:11:09.246958 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.246914 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht"] Oct 31 16:11:09.247588 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.247566 4399 cpu_manager.go:414] "containerMap: removing container" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" Oct 31 16:11:09.247682 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.247591 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" Oct 31 16:11:09.247776 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.247760 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="328607f0-636e-4f5b-8e94-c2132ad25fbd" containerName="istio-proxy" Oct 31 16:11:09.247776 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.247774 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6207a0ed-36ad-4ef4-aa6a-074f0bc1b300" containerName="istio-proxy" Oct 31 16:11:09.252210 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.252189 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.303627 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.303601 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9e4bd6cf-openshift-default-dockercfg-b58z4\"" Oct 31 16:11:09.362639 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.362605 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.362639 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.362641 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.362878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.362664 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.362878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.362730 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.362878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.362795 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.362878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.362819 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.363016 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.362890 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.363016 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.362943 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcnkv\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-kube-api-access-rcnkv\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.363016 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.362998 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.453285 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.453249 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht"] Oct 31 16:11:09.464522 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.464479 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.464522 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.464518 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.464816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.464544 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.464816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.464579 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.464816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.464603 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.464816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.464634 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcnkv\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-kube-api-access-rcnkv\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.464816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.464657 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.464816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.464795 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.465158 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.464823 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.465158 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.465019 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.465158 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.465115 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.465330 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.465266 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.465521 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.465497 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.465597 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.465582 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.466949 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.466926 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.467395 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.467374 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.520804 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.520729 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.521007 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.520981 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcnkv\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-kube-api-access-rcnkv\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.565546 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.565516 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:09.991861 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.991839 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht"] Oct 31 16:11:09.994731 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:11:09.994686 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced266a7_9a88_467f_8dbb_a6775a34e8ee.slice/crio-aef41ce881ccdc2ced59e472bd8e1c68f0236fa8215b43feb5474d80be0e465c WatchSource:0}: Error finding container aef41ce881ccdc2ced59e472bd8e1c68f0236fa8215b43feb5474d80be0e465c: Status 404 returned error can't find the container with id aef41ce881ccdc2ced59e472bd8e1c68f0236fa8215b43feb5474d80be0e465c Oct 31 16:11:09.996883 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.996852 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:11:09.996977 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.996913 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:11:09.996977 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:09.996941 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:11:10.027428 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:10.027400 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" event={"ID":"ced266a7-9a88-467f-8dbb-a6775a34e8ee","Type":"ContainerStarted","Data":"aef41ce881ccdc2ced59e472bd8e1c68f0236fa8215b43feb5474d80be0e465c"} Oct 31 16:11:11.033362 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:11.033327 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" event={"ID":"ced266a7-9a88-467f-8dbb-a6775a34e8ee","Type":"ContainerStarted","Data":"efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3"} Oct 31 16:11:11.565881 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:11.565845 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:11.567511 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:11.567486 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 16:11:11.567647 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:11.567549 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 16:11:12.566154 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:12.566114 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 16:11:12.566579 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:12.566179 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 16:11:13.566263 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:13.566218 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 16:11:13.566736 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:13.566290 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 16:11:14.566228 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:14.566190 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 16:11:14.566445 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:14.566255 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 16:11:15.566182 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:15.566135 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" start-of-body= Oct 31 16:11:15.566404 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:15.566191 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.164:15021/healthz/ready\": dial tcp 10.133.0.164:15021: connect: connection refused" Oct 31 16:11:16.569928 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:16.569894 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:16.570399 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:16.570183 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:16.570955 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:16.570937 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:16.736693 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:16.736638 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" podStartSLOduration=7.736623096 podStartE2EDuration="7.736623096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:11.263049734 +0000 UTC m=+5114.474291147" watchObservedRunningTime="2026-10-31 16:11:16.736623096 +0000 UTC m=+5119.947864507" Oct 31 16:11:26.206343 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:11:26.206306 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:11:39.478436 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.478401 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg"] Oct 31 16:11:39.482213 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.482194 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.552384 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.552359 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.552551 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.552395 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwwn\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-kube-api-access-6dwwn\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.552551 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.552444 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.552551 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.552534 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.552691 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.552551 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.552691 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.552571 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.552691 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.552593 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e012f77b-d22f-4a1e-9277-240f7fbe249a-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.552691 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.552680 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.552868 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.552748 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.593831 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.593793 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg"] Oct 31 16:11:39.653247 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653217 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653247 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653259 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwwn\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-kube-api-access-6dwwn\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653302 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653332 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653356 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653384 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653428 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e012f77b-d22f-4a1e-9277-240f7fbe249a-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653772 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653481 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653772 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653519 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653904 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653792 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.653904 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653877 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-certs\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.654025 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.653945 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-credential-socket\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.654153 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.654129 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-data\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.654345 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.654324 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e012f77b-d22f-4a1e-9277-240f7fbe249a-istiod-ca-cert\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.655724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.655683 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-envoy\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.656137 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.656117 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-podinfo\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.709563 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.709533 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwwn\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-kube-api-access-6dwwn\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.709769 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.709585 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-token\") pod \"e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:39.795763 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:39.795667 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:40.228535 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:40.228506 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg"] Oct 31 16:11:40.231351 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:11:40.231325 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode012f77b_d22f_4a1e_9277_240f7fbe249a.slice/crio-9852a313163b2f766abd45f62b2daa9d70e02df6f9972c73db8ec3c46ee7bbd9 WatchSource:0}: Error finding container 9852a313163b2f766abd45f62b2daa9d70e02df6f9972c73db8ec3c46ee7bbd9: Status 404 returned error can't find the container with id 9852a313163b2f766abd45f62b2daa9d70e02df6f9972c73db8ec3c46ee7bbd9 Oct 31 16:11:40.233525 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:40.233494 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:11:40.233583 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:40.233564 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:11:40.233628 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:40.233594 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:11:41.203048 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:41.203004 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" event={"ID":"e012f77b-d22f-4a1e-9277-240f7fbe249a","Type":"ContainerStarted","Data":"6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2"} Oct 31 16:11:41.203531 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:41.203063 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" event={"ID":"e012f77b-d22f-4a1e-9277-240f7fbe249a","Type":"ContainerStarted","Data":"9852a313163b2f766abd45f62b2daa9d70e02df6f9972c73db8ec3c46ee7bbd9"} Oct 31 16:11:41.371418 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:41.371363 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" podStartSLOduration=2.371347797 podStartE2EDuration="2.371347797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:11:41.367832605 +0000 UTC m=+5144.579074052" watchObservedRunningTime="2026-10-31 16:11:41.371347797 +0000 UTC m=+5144.582589208" Oct 31 16:11:41.796178 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:41.796140 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:41.797887 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:41.797860 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 16:11:41.798038 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:41.797914 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" podUID="e012f77b-d22f-4a1e-9277-240f7fbe249a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 16:11:42.796981 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:42.796946 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 16:11:42.797385 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:42.797012 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" podUID="e012f77b-d22f-4a1e-9277-240f7fbe249a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 16:11:43.796893 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:43.796856 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 16:11:43.797089 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:43.796913 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" podUID="e012f77b-d22f-4a1e-9277-240f7fbe249a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 16:11:44.796236 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:44.796195 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 31 16:11:44.796428 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:44.796266 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" podUID="e012f77b-d22f-4a1e-9277-240f7fbe249a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 31 16:11:45.268831 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:45.268783 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht"] Oct 31 16:11:45.269322 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:45.269037 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" containerID="cri-o://efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3" gracePeriod=30 Oct 31 16:11:45.277533 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:45.277503 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg"] Oct 31 16:11:45.277720 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:45.277681 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" podUID="e012f77b-d22f-4a1e-9277-240f7fbe249a" containerName="istio-proxy" containerID="cri-o://6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2" gracePeriod=30 Oct 31 16:11:46.572960 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:46.572914 4399 patch_prober.go:29] interesting pod/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 16:11:46.573382 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:46.572989 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 16:11:50.531370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.531309 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:50.665278 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.665243 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-socket\") pod \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " Oct 31 16:11:50.665513 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.665292 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-data\") pod \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " Oct 31 16:11:50.665513 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.665329 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-envoy\") pod \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " Oct 31 16:11:50.665513 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.665400 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-token\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-token\") pod \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " Oct 31 16:11:50.665513 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.665478 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-kube-api-access-rcnkv\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-kube-api-access-rcnkv\") pod \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " Oct 31 16:11:50.665828 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.665538 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-credential-socket\") pod \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " Oct 31 16:11:50.666539 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.666491 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-socket" pod "ced266a7-9a88-467f-8dbb-a6775a34e8ee" (UID: "ced266a7-9a88-467f-8dbb-a6775a34e8ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.666759 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.666726 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-data" pod "ced266a7-9a88-467f-8dbb-a6775a34e8ee" (UID: "ced266a7-9a88-467f-8dbb-a6775a34e8ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.667233 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.667193 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-credential-socket" pod "ced266a7-9a88-467f-8dbb-a6775a34e8ee" (UID: "ced266a7-9a88-467f-8dbb-a6775a34e8ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.669079 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.669043 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-certs\") pod \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " Oct 31 16:11:50.669203 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.669092 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-podinfo\") pod \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " Oct 31 16:11:50.669870 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.669575 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-envoy" pod "ced266a7-9a88-467f-8dbb-a6775a34e8ee" (UID: "ced266a7-9a88-467f-8dbb-a6775a34e8ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.669870 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.669591 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-kube-api-access-rcnkv" pod "ced266a7-9a88-467f-8dbb-a6775a34e8ee" (UID: "ced266a7-9a88-467f-8dbb-a6775a34e8ee"). InnerVolumeSpecName "kube-api-access-rcnkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:50.669870 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.669755 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istiod-ca-cert\") pod \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\" (UID: \"ced266a7-9a88-467f-8dbb-a6775a34e8ee\") " Oct 31 16:11:50.669870 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.669772 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-certs" pod "ced266a7-9a88-467f-8dbb-a6775a34e8ee" (UID: "ced266a7-9a88-467f-8dbb-a6775a34e8ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:50.670187 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.670151 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istiod-ca-cert" pod "ced266a7-9a88-467f-8dbb-a6775a34e8ee" (UID: "ced266a7-9a88-467f-8dbb-a6775a34e8ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:11:50.670187 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.670165 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcnkv\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-kube-api-access-rcnkv\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.670187 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.670184 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.670340 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.670198 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.670340 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.670211 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.670340 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.670224 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.670340 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.670236 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.678999 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.678957 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-token" pod "ced266a7-9a88-467f-8dbb-a6775a34e8ee" (UID: "ced266a7-9a88-467f-8dbb-a6775a34e8ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:50.684593 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.684018 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-podinfo" pod "ced266a7-9a88-467f-8dbb-a6775a34e8ee" (UID: "ced266a7-9a88-467f-8dbb-a6775a34e8ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:11:50.771548 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.771504 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.771548 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.771547 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:50.771800 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:50.771562 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ced266a7-9a88-467f-8dbb-a6775a34e8ee-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.257448 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.257409 4399 generic.go:356] "Generic (PLEG): container finished" podID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerID="efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3" exitCode=0 Oct 31 16:11:51.257620 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.257476 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" event={"ID":"ced266a7-9a88-467f-8dbb-a6775a34e8ee","Type":"ContainerDied","Data":"efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3"} Oct 31 16:11:51.257620 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.257536 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" event={"ID":"ced266a7-9a88-467f-8dbb-a6775a34e8ee","Type":"ContainerDied","Data":"aef41ce881ccdc2ced59e472bd8e1c68f0236fa8215b43feb5474d80be0e465c"} Oct 31 16:11:51.257620 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.257477 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht" Oct 31 16:11:51.257620 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.257557 4399 scope.go:122] "RemoveContainer" containerID="efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3" Oct 31 16:11:51.267297 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.267276 4399 scope.go:122] "RemoveContainer" containerID="efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3" Oct 31 16:11:51.267613 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:11:51.267576 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3\": container with ID starting with efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3 not found: ID does not exist" containerID="efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3" Oct 31 16:11:51.267763 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.267619 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3"} err="failed to get container status \"efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3\": rpc error: code = NotFound desc = could not find container \"efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3\": container with ID starting with efb91dc13736db544d0c0d5a037e0e7abfdcb0657a04803798cabc66460e19e3 not found: ID does not exist" Oct 31 16:11:51.469997 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.469970 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht"] Oct 31 16:11:51.520231 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.520201 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-v8hht"] Oct 31 16:11:51.530353 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.530334 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:51.681046 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681016 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-token\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-token\") pod \"e012f77b-d22f-4a1e-9277-240f7fbe249a\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " Oct 31 16:11:51.681477 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681079 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-envoy\") pod \"e012f77b-d22f-4a1e-9277-240f7fbe249a\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " Oct 31 16:11:51.681477 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681184 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-kube-api-access-6dwwn\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-kube-api-access-6dwwn\") pod \"e012f77b-d22f-4a1e-9277-240f7fbe249a\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " Oct 31 16:11:51.681477 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681236 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-certs\") pod \"e012f77b-d22f-4a1e-9277-240f7fbe249a\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " Oct 31 16:11:51.681477 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681271 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-podinfo\") pod \"e012f77b-d22f-4a1e-9277-240f7fbe249a\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " Oct 31 16:11:51.681477 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681313 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e012f77b-d22f-4a1e-9277-240f7fbe249a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e012f77b-d22f-4a1e-9277-240f7fbe249a-istiod-ca-cert\") pod \"e012f77b-d22f-4a1e-9277-240f7fbe249a\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " Oct 31 16:11:51.681477 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681348 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-socket\") pod \"e012f77b-d22f-4a1e-9277-240f7fbe249a\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " Oct 31 16:11:51.681477 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681441 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-credential-socket\") pod \"e012f77b-d22f-4a1e-9277-240f7fbe249a\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " Oct 31 16:11:51.681477 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681475 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-data\") pod \"e012f77b-d22f-4a1e-9277-240f7fbe249a\" (UID: \"e012f77b-d22f-4a1e-9277-240f7fbe249a\") " Oct 31 16:11:51.682153 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681520 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-certs" pod "e012f77b-d22f-4a1e-9277-240f7fbe249a" (UID: "e012f77b-d22f-4a1e-9277-240f7fbe249a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.682153 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681655 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-socket" pod "e012f77b-d22f-4a1e-9277-240f7fbe249a" (UID: "e012f77b-d22f-4a1e-9277-240f7fbe249a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.682153 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.681881 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-credential-socket" pod "e012f77b-d22f-4a1e-9277-240f7fbe249a" (UID: "e012f77b-d22f-4a1e-9277-240f7fbe249a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.682153 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.682050 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-data" pod "e012f77b-d22f-4a1e-9277-240f7fbe249a" (UID: "e012f77b-d22f-4a1e-9277-240f7fbe249a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.682153 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.682058 4399 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.682153 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.682083 4399 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-workload-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.682153 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.682097 4399 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-credential-socket\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.682774 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.682378 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e012f77b-d22f-4a1e-9277-240f7fbe249a-istiod-ca-cert" pod "e012f77b-d22f-4a1e-9277-240f7fbe249a" (UID: "e012f77b-d22f-4a1e-9277-240f7fbe249a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:11:51.683630 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.683598 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-token" pod "e012f77b-d22f-4a1e-9277-240f7fbe249a" (UID: "e012f77b-d22f-4a1e-9277-240f7fbe249a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:51.683783 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.683763 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-kube-api-access-6dwwn" pod "e012f77b-d22f-4a1e-9277-240f7fbe249a" (UID: "e012f77b-d22f-4a1e-9277-240f7fbe249a"). InnerVolumeSpecName "kube-api-access-6dwwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:11:51.683848 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.683806 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-envoy" pod "e012f77b-d22f-4a1e-9277-240f7fbe249a" (UID: "e012f77b-d22f-4a1e-9277-240f7fbe249a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 16:11:51.684805 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.684781 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-podinfo" pod "e012f77b-d22f-4a1e-9277-240f7fbe249a" (UID: "e012f77b-d22f-4a1e-9277-240f7fbe249a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 16:11:51.782888 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.782823 4399 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-data\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.782888 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.782849 4399 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-token\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.782888 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.782858 4399 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-envoy\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.782888 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.782868 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwwn\" (UniqueName: \"kubernetes.io/projected/e012f77b-d22f-4a1e-9277-240f7fbe249a-kube-api-access-6dwwn\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.782888 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.782878 4399 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e012f77b-d22f-4a1e-9277-240f7fbe249a-istio-podinfo\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:51.782888 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:51.782886 4399 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e012f77b-d22f-4a1e-9277-240f7fbe249a-istiod-ca-cert\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:11:52.264470 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.264434 4399 generic.go:356] "Generic (PLEG): container finished" podID="e012f77b-d22f-4a1e-9277-240f7fbe249a" containerID="6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2" exitCode=0 Oct 31 16:11:52.264642 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.264480 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" Oct 31 16:11:52.264642 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.264492 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" event={"ID":"e012f77b-d22f-4a1e-9277-240f7fbe249a","Type":"ContainerDied","Data":"6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2"} Oct 31 16:11:52.264642 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.264532 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg" event={"ID":"e012f77b-d22f-4a1e-9277-240f7fbe249a","Type":"ContainerDied","Data":"9852a313163b2f766abd45f62b2daa9d70e02df6f9972c73db8ec3c46ee7bbd9"} Oct 31 16:11:52.264642 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.264555 4399 scope.go:122] "RemoveContainer" containerID="6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2" Oct 31 16:11:52.276958 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.276933 4399 scope.go:122] "RemoveContainer" containerID="6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2" Oct 31 16:11:52.277231 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:11:52.277210 4399 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2\": container with ID starting with 6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2 not found: ID does not exist" containerID="6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2" Oct 31 16:11:52.277288 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.277241 4399 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2"} err="failed to get container status \"6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2\": rpc error: code = NotFound desc = could not find container \"6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2\": container with ID starting with 6520204d74757f2826459e475f3aaca2cd02fb05b5b62a438b6235465e2e34f2 not found: ID does not exist" Oct 31 16:11:52.426139 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.426108 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg"] Oct 31 16:11:52.475239 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.475208 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9e4bd6cf-openshift-default-ff98cf867-95wwg"] Oct 31 16:11:52.894623 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.894589 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" path="/var/lib/kubelet/pods/ced266a7-9a88-467f-8dbb-a6775a34e8ee/volumes" Oct 31 16:11:52.895024 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:11:52.895015 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e012f77b-d22f-4a1e-9277-240f7fbe249a" path="/var/lib/kubelet/pods/e012f77b-d22f-4a1e-9277-240f7fbe249a/volumes" Oct 31 16:11:56.206199 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:11:56.206162 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:12:08.922671 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.922590 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj"] Oct 31 16:12:08.923059 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.923005 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e012f77b-d22f-4a1e-9277-240f7fbe249a" containerName="istio-proxy" Oct 31 16:12:08.923059 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.923018 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e012f77b-d22f-4a1e-9277-240f7fbe249a" containerName="istio-proxy" Oct 31 16:12:08.923059 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.923053 4399 cpu_manager.go:414] "containerMap: removing container" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" Oct 31 16:12:08.923059 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.923059 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" Oct 31 16:12:08.923197 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.923130 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="ced266a7-9a88-467f-8dbb-a6775a34e8ee" containerName="istio-proxy" Oct 31 16:12:08.923197 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.923139 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e012f77b-d22f-4a1e-9277-240f7fbe249a" containerName="istio-proxy" Oct 31 16:12:08.927839 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.927819 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.937683 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.937657 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.937802 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.937721 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.937863 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.937837 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.937902 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.937883 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.938021 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.938005 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.938100 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.938086 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.938245 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.938226 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.938295 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.938256 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx8vf\" (UniqueName: \"kubernetes.io/projected/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-kube-api-access-sx8vf\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.938334 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.938313 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:08.979457 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:08.979433 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" Oct 31 16:12:09.039008 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.038974 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039156 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039017 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039156 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039036 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx8vf\" (UniqueName: \"kubernetes.io/projected/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-kube-api-access-sx8vf\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039156 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039061 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039156 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039104 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039156 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039125 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039156 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039154 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039516 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039184 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039516 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039273 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039516 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039466 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039846 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039783 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.039846 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.039829 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.040200 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.040023 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.040200 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.040026 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.042343 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.042313 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.042456 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.042432 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.094875 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.094854 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.095140 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.095120 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx8vf\" (UniqueName: \"kubernetes.io/projected/f96f57e6-3024-4ef2-b7fb-8a49086ce52d-kube-api-access-sx8vf\") pod \"maas-default-gateway-openshift-default-67bf986f99-hvzpj\" (UID: \"f96f57e6-3024-4ef2-b7fb-8a49086ce52d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.133112 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.133087 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj"] Oct 31 16:12:09.240517 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.240493 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:09.663541 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.663517 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj"] Oct 31 16:12:09.665786 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:12:09.665760 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96f57e6_3024_4ef2_b7fb_8a49086ce52d.slice/crio-40992fbacf5147acf08571c51d6a502b98be154d22c0ceb34e32cfedc9a2343d WatchSource:0}: Error finding container 40992fbacf5147acf08571c51d6a502b98be154d22c0ceb34e32cfedc9a2343d: Status 404 returned error can't find the container with id 40992fbacf5147acf08571c51d6a502b98be154d22c0ceb34e32cfedc9a2343d Oct 31 16:12:09.667904 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.667870 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:12:09.668029 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.667949 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:12:09.668029 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:09.667997 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:12:10.370390 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:10.370352 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" event={"ID":"f96f57e6-3024-4ef2-b7fb-8a49086ce52d","Type":"ContainerStarted","Data":"57e30b612199d5172068e1dce2004e57f71804b5da53a7fa2c8a068fec731ae9"} Oct 31 16:12:10.370958 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:10.370417 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" event={"ID":"f96f57e6-3024-4ef2-b7fb-8a49086ce52d","Type":"ContainerStarted","Data":"40992fbacf5147acf08571c51d6a502b98be154d22c0ceb34e32cfedc9a2343d"} Oct 31 16:12:10.537959 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:10.537905 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podStartSLOduration=2.537890553 podStartE2EDuration="2.537890553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:12:10.535527426 +0000 UTC m=+5173.746768860" watchObservedRunningTime="2026-10-31 16:12:10.537890553 +0000 UTC m=+5173.749131965" Oct 31 16:12:11.265112 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:11.265072 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:11.267447 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:11.267417 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 16:12:11.267578 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:11.267471 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 16:12:12.260722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:12.256073 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 16:12:12.260722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:12.256148 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 16:12:13.252723 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:13.242406 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 16:12:13.252723 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:13.242488 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 16:12:14.245129 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:14.245092 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 16:12:14.249817 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:14.249775 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 16:12:15.255763 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:15.255725 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 16:12:15.256401 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:15.256370 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 16:12:16.241062 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:16.241033 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 16:12:16.241265 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:16.241090 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 16:12:17.241124 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:17.241092 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 31 16:12:17.241508 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:17.241154 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 31 16:12:18.879030 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:18.878991 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:18.879508 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:18.879417 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:18.880563 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:12:18.880542 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" Oct 31 16:12:26.206502 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:12:26.206468 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:12:56.205791 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:12:56.205757 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:13:09.412819 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.412783 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr"] Oct 31 16:13:09.416951 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.416928 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.514759 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.514730 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09561dd-b9cb-49e2-841d-54d528822afb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.514933 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.514772 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.514933 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.514813 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09561dd-b9cb-49e2-841d-54d528822afb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.515022 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.514933 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.515022 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.514968 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.515022 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.514997 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09561dd-b9cb-49e2-841d-54d528822afb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.515167 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.515027 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6c2\" (UniqueName: \"kubernetes.io/projected/c09561dd-b9cb-49e2-841d-54d528822afb-kube-api-access-gb6c2\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.515167 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.515066 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.515167 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.515105 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.616740 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.616667 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.616925 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.616745 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.616925 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.616834 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09561dd-b9cb-49e2-841d-54d528822afb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.616925 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.616874 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.616925 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.616907 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09561dd-b9cb-49e2-841d-54d528822afb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.617161 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.616963 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.617161 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.617003 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.617161 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.617035 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09561dd-b9cb-49e2-841d-54d528822afb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.617161 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.617067 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6c2\" (UniqueName: \"kubernetes.io/projected/c09561dd-b9cb-49e2-841d-54d528822afb-kube-api-access-gb6c2\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.617161 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.617105 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.617655 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.617384 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.618559 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.618039 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c09561dd-b9cb-49e2-841d-54d528822afb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.618559 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.618311 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.618559 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.618414 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.619480 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.619455 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c09561dd-b9cb-49e2-841d-54d528822afb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.619855 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.619812 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c09561dd-b9cb-49e2-841d-54d528822afb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.620725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.620685 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr"] Oct 31 16:13:09.679014 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.678950 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c09561dd-b9cb-49e2-841d-54d528822afb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.679014 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.678997 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6c2\" (UniqueName: \"kubernetes.io/projected/c09561dd-b9cb-49e2-841d-54d528822afb-kube-api-access-gb6c2\") pod \"maas-default-gateway-openshift-default-67bf986f99-vfnzr\" (UID: \"c09561dd-b9cb-49e2-841d-54d528822afb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:09.729827 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:09.729802 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:10.156432 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:10.156400 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr"] Oct 31 16:13:10.158927 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:13:10.158898 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc09561dd_b9cb_49e2_841d_54d528822afb.slice/crio-4fa15c3a8685db37fd4ddffd4e8cd1aff50c237b71ba04afc1d0d5187e624614 WatchSource:0}: Error finding container 4fa15c3a8685db37fd4ddffd4e8cd1aff50c237b71ba04afc1d0d5187e624614: Status 404 returned error can't find the container with id 4fa15c3a8685db37fd4ddffd4e8cd1aff50c237b71ba04afc1d0d5187e624614 Oct 31 16:13:10.161205 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:10.161176 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:13:10.161300 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:10.161234 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:13:10.161300 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:10.161267 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:13:10.667630 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:10.667592 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" event={"ID":"c09561dd-b9cb-49e2-841d-54d528822afb","Type":"ContainerStarted","Data":"d45da89e6c871239751059ff330d60134154e0debfff2fe0a79256623d5c0d7e"} Oct 31 16:13:10.668187 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:10.667657 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" event={"ID":"c09561dd-b9cb-49e2-841d-54d528822afb","Type":"ContainerStarted","Data":"4fa15c3a8685db37fd4ddffd4e8cd1aff50c237b71ba04afc1d0d5187e624614"} Oct 31 16:13:10.730848 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:10.730821 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:10.833222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:10.833171 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podStartSLOduration=1.833157571 podStartE2EDuration="1.833157571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:10.831088146 +0000 UTC m=+5234.042329559" watchObservedRunningTime="2026-10-31 16:13:10.833157571 +0000 UTC m=+5234.044398982" Oct 31 16:13:11.732611 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:11.732575 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 16:13:11.733048 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:11.732633 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 16:13:12.731039 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:12.731005 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 16:13:12.731224 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:12.731064 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 16:13:13.730101 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:13.730068 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 16:13:13.730512 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:13.730129 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 16:13:14.730869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:14.730834 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 16:13:14.731284 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:14.730895 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 16:13:15.731233 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:15.731194 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 16:13:15.731643 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:15.731257 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 16:13:16.730139 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:16.730105 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 16:13:16.730316 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:16.730164 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 16:13:17.730468 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:17.730432 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body= Oct 31 16:13:17.730889 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:17.730495 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" Oct 31 16:13:19.730671 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:19.730631 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:19.731094 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:19.730690 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:20.771878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:20.771832 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:20.772349 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:20.771913 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:21.813905 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:21.813861 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:21.814432 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:21.813920 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:22.818830 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:22.818791 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:22.819550 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:22.819519 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:23.860908 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:23.860869 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:23.861554 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:23.861519 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:24.524017 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.523984 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z"] Oct 31 16:13:24.527838 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.527812 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.595227 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.595177 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.595421 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.595236 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.595421 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.595270 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98k29\" (UniqueName: \"kubernetes.io/projected/116d08e1-3b25-43fb-92e9-2b960cd2e50d-kube-api-access-98k29\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.595421 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.595362 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.595421 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.595397 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.595682 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.595426 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.595682 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.595520 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.595682 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.595571 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.595682 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.595597 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.632084 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.632054 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z"] Oct 31 16:13:24.697278 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.697241 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.697487 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.697298 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.697487 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.697332 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.697487 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.697422 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.697487 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.697458 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.697487 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.697483 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.697864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.697512 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.697864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.697546 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.697864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.697579 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98k29\" (UniqueName: \"kubernetes.io/projected/116d08e1-3b25-43fb-92e9-2b960cd2e50d-kube-api-access-98k29\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.699180 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.698355 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.699180 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.698803 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.699180 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.699132 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.699672 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.699645 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.700019 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.699996 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.709027 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.709001 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.709569 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.709537 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.730440 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.730419 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:24.754617 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.754589 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/116d08e1-3b25-43fb-92e9-2b960cd2e50d-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.754841 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.754599 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98k29\" (UniqueName: \"kubernetes.io/projected/116d08e1-3b25-43fb-92e9-2b960cd2e50d-kube-api-access-98k29\") pod \"maas-default-gateway-openshift-default-67bf986f99-xgc5z\" (UID: \"116d08e1-3b25-43fb-92e9-2b960cd2e50d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.847013 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.846919 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:24.907890 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.907849 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:24.908376 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:24.907901 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:25.225636 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.225602 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:25.226525 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.226503 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" Oct 31 16:13:25.289008 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:13:25.288977 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116d08e1_3b25_43fb_92e9_2b960cd2e50d.slice/crio-0af8859bbec8cf79e5c34e3e41336cbe9b6f4014dfc849b161aa4ec1a94a75cb WatchSource:0}: Error finding container 0af8859bbec8cf79e5c34e3e41336cbe9b6f4014dfc849b161aa4ec1a94a75cb: Status 404 returned error can't find the container with id 0af8859bbec8cf79e5c34e3e41336cbe9b6f4014dfc849b161aa4ec1a94a75cb Oct 31 16:13:25.291127 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.291097 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:13:25.291244 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.291179 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:13:25.291244 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.291222 4399 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 16:13:25.328674 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.328629 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z"] Oct 31 16:13:25.772993 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.772883 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" event={"ID":"116d08e1-3b25-43fb-92e9-2b960cd2e50d","Type":"ContainerStarted","Data":"540538c12c13a2971dd8b58d46606c2204559f7a64007b3ed1d95f587cffac70"} Oct 31 16:13:25.772993 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.772944 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" event={"ID":"116d08e1-3b25-43fb-92e9-2b960cd2e50d","Type":"ContainerStarted","Data":"0af8859bbec8cf79e5c34e3e41336cbe9b6f4014dfc849b161aa4ec1a94a75cb"} Oct 31 16:13:25.855847 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.855805 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:25.887991 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:25.887936 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podStartSLOduration=1.8879199519999998 podStartE2EDuration="1.887919952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:13:25.886085529 +0000 UTC m=+5249.097326941" watchObservedRunningTime="2026-10-31 16:13:25.887919952 +0000 UTC m=+5249.099161363" Oct 31 16:13:26.207173 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:13:26.207124 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:13:26.865725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:26.864823 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 16:13:26.865725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:26.864896 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 16:13:27.862724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:27.862660 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 16:13:27.863256 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:27.862774 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 16:13:28.850724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:28.848322 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 16:13:28.850724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:28.848377 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 16:13:29.859730 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:29.854161 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 16:13:29.859730 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:29.854219 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 16:13:30.861723 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:30.861095 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 16:13:30.861723 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:30.861154 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 16:13:31.847435 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:31.847393 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 16:13:31.847630 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:31.847460 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 16:13:32.847916 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:32.847880 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body= Oct 31 16:13:32.848574 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:32.848526 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" Oct 31 16:13:34.904723 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:34.898816 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:34.904723 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:34.898895 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:34.926909 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:34.926866 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:34.927069 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:34.926924 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:35.900271 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:35.900238 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:35.900457 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:35.900292 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:36.941922 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:36.941874 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:36.942455 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:36.941943 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:37.983936 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:37.983853 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:37.983936 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:37.983923 4399 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:13:38.248111 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:38.248027 4399 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:38.248333 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:38.248313 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:38.249048 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:38.249028 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" Oct 31 16:13:56.270723 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:13:56.267162 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:13:56.271242 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:56.270792 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:13:56.271242 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:13:56.270848 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:14:26.211826 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:14:26.211785 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:14:34.926871 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:14:34.926829 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:14:34.927387 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:14:34.926901 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:14:56.206528 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:14:56.206492 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:15:26.206726 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:15:26.206660 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:15:56.231642 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:15:56.231595 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:16:08.503508 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:16:08.503460 4399 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" podUID="dc41cca5aaa38cfa8c691e2d5baf2778" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 31 16:16:26.210773 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:16:26.210733 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:16:56.207155 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:16:56.207120 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:17:26.206688 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:17:26.206652 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:17:56.206812 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:17:56.206775 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:18:26.206749 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:18:26.206685 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:18:56.209211 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:18:56.209137 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:19:26.240659 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:19:26.240620 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:19:26.268727 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:19:26.267830 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:19:26.268727 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:19:26.267924 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:19:41.273958 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:19:41.273852 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:19:41.274715 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:19:41.274642 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:19:56.207004 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:19:56.206963 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:20:26.209557 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:20:26.209522 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:20:36.192957 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:36.192923 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-pth7n"] Oct 31 16:20:41.194071 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.194027 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" podUID="8a05f09f-b51e-4f82-b934-7a697b365d80" containerName="payload-pre-processing" containerID="cri-o://d575509ca9f12bf0c80e8a0f45d0822d74d8269f8c78792ab710e2f48d15b7be" gracePeriod=25 Oct 31 16:20:41.356863 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.356824 4399 generic.go:356] "Generic (PLEG): container finished" podID="8a05f09f-b51e-4f82-b934-7a697b365d80" containerID="d575509ca9f12bf0c80e8a0f45d0822d74d8269f8c78792ab710e2f48d15b7be" exitCode=0 Oct 31 16:20:41.357059 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.356907 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" event={"ID":"8a05f09f-b51e-4f82-b934-7a697b365d80","Type":"ContainerDied","Data":"d575509ca9f12bf0c80e8a0f45d0822d74d8269f8c78792ab710e2f48d15b7be"} Oct 31 16:20:41.445581 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.445561 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 16:20:41.594445 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.594410 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a05f09f-b51e-4f82-b934-7a697b365d80-kube-api-access-t6g64\" (UniqueName: \"kubernetes.io/projected/8a05f09f-b51e-4f82-b934-7a697b365d80-kube-api-access-t6g64\") pod \"8a05f09f-b51e-4f82-b934-7a697b365d80\" (UID: \"8a05f09f-b51e-4f82-b934-7a697b365d80\") " Oct 31 16:20:41.594606 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.594451 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a05f09f-b51e-4f82-b934-7a697b365d80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a05f09f-b51e-4f82-b934-7a697b365d80-plugins-config-volume\") pod \"8a05f09f-b51e-4f82-b934-7a697b365d80\" (UID: \"8a05f09f-b51e-4f82-b934-7a697b365d80\") " Oct 31 16:20:41.594885 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.594864 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a05f09f-b51e-4f82-b934-7a697b365d80-plugins-config-volume" pod "8a05f09f-b51e-4f82-b934-7a697b365d80" (UID: "8a05f09f-b51e-4f82-b934-7a697b365d80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:20:41.596871 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.596847 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a05f09f-b51e-4f82-b934-7a697b365d80-kube-api-access-t6g64" pod "8a05f09f-b51e-4f82-b934-7a697b365d80" (UID: "8a05f09f-b51e-4f82-b934-7a697b365d80"). InnerVolumeSpecName "kube-api-access-t6g64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:20:41.696307 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.696232 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6g64\" (UniqueName: \"kubernetes.io/projected/8a05f09f-b51e-4f82-b934-7a697b365d80-kube-api-access-t6g64\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:20:41.696307 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:41.696262 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a05f09f-b51e-4f82-b934-7a697b365d80-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:20:42.362786 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:42.362755 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" Oct 31 16:20:42.363239 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:42.362754 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-pth7n" event={"ID":"8a05f09f-b51e-4f82-b934-7a697b365d80","Type":"ContainerDied","Data":"b8874d02bf2e30f60270788cc4f1955307d32b8d80aa74341ac733f0d7f6205f"} Oct 31 16:20:42.363239 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:42.362899 4399 scope.go:122] "RemoveContainer" containerID="d575509ca9f12bf0c80e8a0f45d0822d74d8269f8c78792ab710e2f48d15b7be" Oct 31 16:20:42.523792 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:42.523756 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-pth7n"] Oct 31 16:20:42.573462 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:42.573429 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-pth7n"] Oct 31 16:20:42.895235 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:42.895202 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a05f09f-b51e-4f82-b934-7a697b365d80" path="/var/lib/kubelet/pods/8a05f09f-b51e-4f82-b934-7a697b365d80/volumes" Oct 31 16:20:50.263511 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:50.263476 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-kzfcq"] Oct 31 16:20:55.264188 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.264136 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" podUID="5aac9441-7f91-49f3-9472-8fcd34fec615" containerName="payload-processing" containerID="cri-o://7199a2c36b9c460cce12c8aad826a027ef465736be15d4854b65fece7e5038f8" gracePeriod=25 Oct 31 16:20:55.428987 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.428955 4399 generic.go:356] "Generic (PLEG): container finished" podID="5aac9441-7f91-49f3-9472-8fcd34fec615" containerID="7199a2c36b9c460cce12c8aad826a027ef465736be15d4854b65fece7e5038f8" exitCode=0 Oct 31 16:20:55.429142 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.429033 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" event={"ID":"5aac9441-7f91-49f3-9472-8fcd34fec615","Type":"ContainerDied","Data":"7199a2c36b9c460cce12c8aad826a027ef465736be15d4854b65fece7e5038f8"} Oct 31 16:20:55.518994 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.518934 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 16:20:55.627087 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.627064 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aac9441-7f91-49f3-9472-8fcd34fec615-kube-api-access-k2f6j\" (UniqueName: \"kubernetes.io/projected/5aac9441-7f91-49f3-9472-8fcd34fec615-kube-api-access-k2f6j\") pod \"5aac9441-7f91-49f3-9472-8fcd34fec615\" (UID: \"5aac9441-7f91-49f3-9472-8fcd34fec615\") " Oct 31 16:20:55.627209 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.627101 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aac9441-7f91-49f3-9472-8fcd34fec615-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aac9441-7f91-49f3-9472-8fcd34fec615-plugins-config-volume\") pod \"5aac9441-7f91-49f3-9472-8fcd34fec615\" (UID: \"5aac9441-7f91-49f3-9472-8fcd34fec615\") " Oct 31 16:20:55.627500 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.627480 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aac9441-7f91-49f3-9472-8fcd34fec615-plugins-config-volume" pod "5aac9441-7f91-49f3-9472-8fcd34fec615" (UID: "5aac9441-7f91-49f3-9472-8fcd34fec615"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:20:55.629322 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.629299 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aac9441-7f91-49f3-9472-8fcd34fec615-kube-api-access-k2f6j" pod "5aac9441-7f91-49f3-9472-8fcd34fec615" (UID: "5aac9441-7f91-49f3-9472-8fcd34fec615"). InnerVolumeSpecName "kube-api-access-k2f6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:20:55.728864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.728826 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2f6j\" (UniqueName: \"kubernetes.io/projected/5aac9441-7f91-49f3-9472-8fcd34fec615-kube-api-access-k2f6j\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:20:55.728864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:55.728854 4399 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5aac9441-7f91-49f3-9472-8fcd34fec615-plugins-config-volume\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:20:56.212256 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:20:56.212217 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:20:56.434747 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:56.434692 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" event={"ID":"5aac9441-7f91-49f3-9472-8fcd34fec615","Type":"ContainerDied","Data":"a88356f09eb4b0bdf2b4553d2ccfbcad1bc9823ea5e61ed953350eaa2a0e70e0"} Oct 31 16:20:56.434747 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:56.434738 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-kzfcq" Oct 31 16:20:56.435301 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:56.434805 4399 scope.go:122] "RemoveContainer" containerID="7199a2c36b9c460cce12c8aad826a027ef465736be15d4854b65fece7e5038f8" Oct 31 16:20:56.596044 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:56.595971 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-kzfcq"] Oct 31 16:20:56.645207 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:56.645173 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-kzfcq"] Oct 31 16:20:56.895065 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:20:56.894993 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aac9441-7f91-49f3-9472-8fcd34fec615" path="/var/lib/kubelet/pods/5aac9441-7f91-49f3-9472-8fcd34fec615/volumes" Oct 31 16:21:26.206065 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:21:26.206027 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:21:56.206833 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:21:56.206796 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:22:24.853125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:22:24.853036 4399 generic.go:356] "Generic (PLEG): container finished" podID="bd10b679-de31-4629-8256-cf75f52e76c4" containerID="ab617043ec6d82e2a3cc7d78beefc6eba6f14581e47631a7ce7d875beef182a6" exitCode=1 Oct 31 16:22:24.853125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:22:24.853111 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" event={"ID":"bd10b679-de31-4629-8256-cf75f52e76c4","Type":"ContainerDied","Data":"ab617043ec6d82e2a3cc7d78beefc6eba6f14581e47631a7ce7d875beef182a6"} Oct 31 16:22:24.853720 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:22:24.853553 4399 scope.go:122] "RemoveContainer" containerID="ab617043ec6d82e2a3cc7d78beefc6eba6f14581e47631a7ce7d875beef182a6" Oct 31 16:22:24.853995 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:22:24.853976 4399 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 16:22:25.771609 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:22:25.771581 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 31 16:22:25.859767 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:22:25.859727 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" event={"ID":"bd10b679-de31-4629-8256-cf75f52e76c4","Type":"ContainerStarted","Data":"b499164952257e8a34d8d5957134108065aabe4e1b6109be95c393016d9462d5"} Oct 31 16:22:25.860146 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:22:25.859991 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 16:22:26.206193 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:22:26.206159 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:22:36.867166 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:22:36.867132 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-78f488d89-cv2qm" Oct 31 16:22:56.206311 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:22:56.206273 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:23:19.922847 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:23:19.922814 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:23:19.923403 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:23:19.923370 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:23:26.206321 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:23:26.206284 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:23:56.206745 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:23:56.206678 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:24:19.922858 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:24:19.922819 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:24:19.923244 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:24:19.922882 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:26.206038 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:24:26.206001 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:24:49.924893 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:24:49.924852 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:24:49.925284 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:24:49.924919 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:24:56.207007 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:24:56.206969 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:25:26.206007 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:25:26.205973 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:25:41.229315 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:25:41.229216 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:25:41.229315 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:25:41.229285 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:25:56.205719 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:25:56.205658 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:26:26.206098 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:26:26.206055 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:26:56.206840 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:26:56.206805 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:27:26.205987 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:27:26.205952 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:27:56.205574 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:27:56.205536 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:28:26.207093 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:28:26.207056 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:28:56.208593 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:28:56.208554 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:29:26.206401 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:29:26.206353 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:29:53.661555 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:29:53.661514 4399 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-137-160.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 16:29:56.209598 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:29:56.209558 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:30:03.662018 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:30:03.661974 4399 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-137-160.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 16:30:08.503590 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:08.503479 4399 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal" podUID="dc41cca5aaa38cfa8c691e2d5baf2778" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 16:30:13.662248 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:30:13.662201 4399 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-137-160.us-west-1.compute.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 16:30:23.663412 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:30:23.663364 4399 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-137-160.us-west-1.compute.internal?timeout=10s\": context deadline exceeded" Oct 31 16:30:25.856606 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856565 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856609 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856619 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856636 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856680 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qrk96\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856751 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856762 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856753 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856791 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856808 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856810 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856834 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856852 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856848 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:30:25.856861 4399 kubelet_node_status.go:524] "Error updating node status, will retry" err="failed to patch status \"{\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"type\\\":\\\"DiskPressure\\\"},{\\\"type\\\":\\\"PIDPressure\\\"},{\\\"type\\\":\\\"Ready\\\"}],\\\"conditions\\\":[{\\\"lastHeartbeatTime\\\":\\\"2026-10-31T16:30:25Z\\\",\\\"type\\\":\\\"MemoryPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-31T16:30:25Z\\\",\\\"type\\\":\\\"DiskPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-31T16:30:25Z\\\",\\\"type\\\":\\\"PIDPressure\\\"},{\\\"lastHeartbeatTime\\\":\\\"2026-10-31T16:30:25Z\\\",\\\"type\\\":\\\"Ready\\\"}],\\\"runtimeHandlers\\\":[{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"runc\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"crun\\\"},{\\\"features\\\":{\\\"recursiveReadOnlyMounts\\\":true,\\\"userNamespaces\\\":true},\\\"name\\\":\\\"\\\"}]}}\" for node \"ip-10-0-137-160.us-west-1.compute.internal\": Patch \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/api/v1/nodes/ip-10-0-137-160.us-west-1.compute.internal/status?timeout=10s\": http2: client connection lost" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856890 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856901 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856917 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856934 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856934 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856967 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.856987 4399 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857026 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857042 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857058 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857066 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857074 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-78mefrvckg941\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857104 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fspng\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857110 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857125 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857133 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.857134 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857141 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857168 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6xfnp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857176 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857201 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857235 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857243 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857251 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857267 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857275 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857282 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857243 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857290 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"downloads-dockercfg-6g8xr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857305 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857340 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857344 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857342 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857359 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857370 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857380 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857391 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857414 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857412 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857432 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jwmrt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857442 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vjkjf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857452 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857477 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857480 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857490 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857482 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857493 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857517 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857535 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-z4lnh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857534 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857555 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857559 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857575 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857600 4399 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857616 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857620 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857615 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857638 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857655 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857653 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857651 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857676 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857706 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1aauuj6ttrpu2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857688 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857719 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857729 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857776 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857785 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857785 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-t5rnb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857797 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lr8w6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857785 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857804 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857793 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857825 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857833 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857819 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwnt5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857806 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857853 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857864 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:30:25.857865 4399 controller.go:251] "Failed to update lease" err="Put \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-137-160.us-west-1.compute.internal?timeout=10s\": http2: client connection lost" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857892 4399 controller.go:171] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857879 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857899 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857905 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857915 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857900 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857871 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857937 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857959 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-m95mw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857959 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857978 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857978 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857993 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.857998 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858001 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858005 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-29hlw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858017 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858028 4399 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858035 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858046 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858057 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858065 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858071 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858069 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m4dnq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858060 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858087 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858087 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858100 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xsph8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858105 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858101 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.858678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858103 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858118 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858118 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k6kcz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858131 4399 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858112 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858131 4399 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858149 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nkdvg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858156 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858164 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858173 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858190 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-m92fm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858166 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858212 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g79w8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858215 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858245 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858251 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858252 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858262 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858270 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rxbg7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858275 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-96tvm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:30:25.858172 4399 event.go:368] "Unable to write event (may retry after sleeping)" err="Post \"https://a512747cf7570402090f1cd78c56a05b-54c583efd5dcf322.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/kube-system/events\": http2: client connection lost" event="&Event{ObjectMeta:{kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal.18e3aa32093ae6ec kube-system 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Pod,Namespace:kube-system,Name:kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal,UID:dc41cca5aaa38cfa8c691e2d5baf2778,APIVersion:v1,ResourceVersion:,FieldPath:spec.containers{haproxy},},Reason:Unhealthy,Message:Liveness probe failed: Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers),Source:EventSource{Component:kubelet,Host:ip-10-0-137-160.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 16:30:08.503531244 +0000 UTC m=+6251.714772647,LastTimestamp:2026-10-31 16:30:08.503531244 +0000 UTC m=+6251.714772647,Count:1,Type:Warning,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-160.us-west-1.compute.internal,}" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858287 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858301 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858308 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858328 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858334 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-89zxn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858334 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858348 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858365 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dtk5s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858367 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858378 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858378 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858369 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858392 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858380 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858392 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858406 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858414 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858418 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858419 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858421 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:25.861680 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:25.858435 4399 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Oct 31 16:30:26.206011 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:30:26.205980 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:30:26.709854 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.709828 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 31 16:30:26.759273 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.759249 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 16:30:26.759526 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.759507 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xsph8\"" Oct 31 16:30:26.759780 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.759769 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 16:30:26.759863 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.759805 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 16:30:26.759999 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.759983 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 31 16:30:26.760479 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.760467 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 16:30:26.760524 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.760469 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-fvjmg\"" Oct 31 16:30:26.858937 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.858915 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 16:30:26.858937 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.858922 4399 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:30:26.859328 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.858943 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 16:30:26.859328 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.859149 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 16:30:26.908783 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.908760 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 16:30:26.908932 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.908823 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 16:30:26.908932 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.908870 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 16:30:26.909034 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.908998 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 16:30:26.909034 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.909027 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 16:30:26.909145 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.909065 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-6g8xr\"" Oct 31 16:30:26.909145 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.909090 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 16:30:26.909145 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.909088 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-89zxn\"" Oct 31 16:30:26.909145 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.909085 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-n7mq6\"" Oct 31 16:30:26.909291 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.909284 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 16:30:26.958610 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.958590 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 16:30:26.958769 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.958677 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 16:30:26.958769 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.958757 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 16:30:26.958907 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.958777 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sv6vw\"" Oct 31 16:30:26.958907 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.958822 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-lr8w6\"" Oct 31 16:30:26.959023 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.958942 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzf6\"" Oct 31 16:30:26.959023 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.958948 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8678r\"" Oct 31 16:30:26.959105 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.959087 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rxbg7\"" Oct 31 16:30:26.959162 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.959148 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 16:30:26.959241 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.959224 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 16:30:26.959533 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.959505 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 16:30:26.959632 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.959595 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-78mefrvckg941\"" Oct 31 16:30:26.959726 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.959636 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 16:30:26.960018 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.959963 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 16:30:26.960178 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.959972 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 16:30:26.960178 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.959994 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 16:30:26.960178 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.960010 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 16:30:26.960366 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.960037 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 16:30:26.960366 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.960050 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 16:30:26.960366 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.960138 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 16:30:26.960366 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.960162 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 16:30:26.960366 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:26.960245 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-z4lnh\"" Oct 31 16:30:27.008327 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.008305 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.008327 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.008322 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 16:30:27.008510 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.008328 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1aauuj6ttrpu2\"" Oct 31 16:30:27.008510 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.008453 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 16:30:27.009465 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009436 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 16:30:27.009581 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009475 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 16:30:27.009581 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009550 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.009742 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009583 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.009742 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009607 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 16:30:27.009742 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009610 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-vjkjf\"" Oct 31 16:30:27.009742 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009626 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 16:30:27.009742 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009630 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.009742 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009663 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.009742 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009742 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pb58t\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009775 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jwmrt\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009844 4399 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009861 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-b5qzb\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009864 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009901 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-96tvm\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009911 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009926 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009941 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009931 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.009974 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.010078 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 31 16:30:27.010125 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.010127 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-qrk96\"" Oct 31 16:30:27.010544 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.010175 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 16:30:27.010544 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.010259 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fspng\"" Oct 31 16:30:27.010544 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.010283 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-29hlw\"" Oct 31 16:30:27.018348 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.018334 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.019708 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.019687 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 16:30:27.044656 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.044639 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 16:30:27.056292 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.056275 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.068735 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.068716 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pj7g4\"" Oct 31 16:30:27.069789 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.069774 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 16:30:27.072273 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.072246 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 16:30:27.073297 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.073281 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.092713 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.092684 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 16:30:27.096343 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.096324 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.097363 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.097348 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7rzhx\"" Oct 31 16:30:27.115202 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.115186 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 31 16:30:27.121843 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.121820 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.123557 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.123540 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 16:30:27.126878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.126853 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p6pvz\"" Oct 31 16:30:27.134141 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.134124 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-k6kcz\"" Oct 31 16:30:27.136275 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.136262 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 16:30:27.140297 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.140276 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.144750 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.144731 4399 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:30:27.148768 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.148752 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 16:30:27.153078 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.153054 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.153078 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.153055 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 16:30:27.155094 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.155079 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rjd9l\"" Oct 31 16:30:27.171569 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.171550 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.172113 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.172095 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 16:30:27.175034 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.175021 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 16:30:27.184588 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.184570 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-dtk5s\"" Oct 31 16:30:27.191074 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.191058 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 16:30:27.198604 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.198587 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 16:30:27.200260 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.200248 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khnf8\"" Oct 31 16:30:27.202396 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.202381 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-6xfnp\"" Oct 31 16:30:27.204660 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.204647 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.220278 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.220225 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2z8r6\"" Oct 31 16:30:27.221264 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.221250 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-t5rnb\"" Oct 31 16:30:27.221900 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.221885 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.234769 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.234750 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 16:30:27.242663 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.242640 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 16:30:27.242760 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.242672 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-gwnt5\"" Oct 31 16:30:27.250412 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.250398 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hrzk8\"" Oct 31 16:30:27.252786 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.252765 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 16:30:27.263382 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.263359 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.263469 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.263395 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 16:30:27.265957 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.265938 4399 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:30:27.268310 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.268296 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 16:30:27.283076 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.283057 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 16:30:27.283976 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.283958 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 16:30:27.291587 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.291573 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-z98bh\"" Oct 31 16:30:27.295268 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.295255 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 16:30:27.301200 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.301188 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 16:30:27.303097 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.303084 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.315605 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.315590 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-m92fm\"" Oct 31 16:30:27.317489 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.317475 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.319816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.319803 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.329344 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.329331 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 16:30:27.331631 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.331617 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.331934 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.331917 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.333586 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.333574 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 16:30:27.349957 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.349941 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 16:30:27.352689 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.352676 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 16:30:27.355590 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.355577 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 16:30:27.366073 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.366059 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 31 16:30:27.376924 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.376907 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.382320 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.382301 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-nxt4l\"" Oct 31 16:30:27.385101 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.385086 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tbgjp\"" Oct 31 16:30:27.391263 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.391243 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 16:30:27.397080 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.397063 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 16:30:27.400196 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.400182 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.404221 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.404205 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 16:30:27.404221 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.404215 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 16:30:27.405252 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.405237 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 16:30:27.407102 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.407089 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-m4dnq\"" Oct 31 16:30:27.410258 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.410245 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.412423 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.412408 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 16:30:27.415930 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.415918 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-g79w8\"" Oct 31 16:30:27.422681 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.422668 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.422762 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.422734 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-nkdvg\"" Oct 31 16:30:27.428538 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.428525 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 16:30:27.440158 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.440141 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.443288 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.443275 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 16:30:27.444342 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.444329 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 16:30:27.446366 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.446354 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.451981 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.451968 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 16:30:27.453892 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.453875 4399 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 16:30:27.463940 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.463926 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 16:30:27.465905 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.465891 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 16:30:27.471648 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.471612 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 16:30:27.472615 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.472602 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 16:30:27.493745 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.493726 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 16:30:27.507620 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:30:27.507599 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-m95mw\"" Oct 31 16:30:56.205984 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:30:56.205949 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:31:26.206441 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:31:26.206405 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:31:56.205948 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:31:56.205858 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:32:26.206176 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:32:26.206070 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:32:41.227329 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:32:41.227283 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:32:41.227852 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:32:41.227349 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:32:56.206174 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:32:56.206133 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:33:11.269789 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:11.268789 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:33:11.269789 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:11.268868 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:33:14.127210 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:14.127162 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-g59d7"] Oct 31 16:33:14.128097 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:14.127411 4399 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" podUID="e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" containerName="maas-api" containerID="cri-o://a7130cb15d7212afe92be879b96e7a54013535f8b517cecb47f357f1f4b77442" gracePeriod=30 Oct 31 16:33:15.333354 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.333318 4399 generic.go:356] "Generic (PLEG): container finished" podID="e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" containerID="a7130cb15d7212afe92be879b96e7a54013535f8b517cecb47f357f1f4b77442" exitCode=0 Oct 31 16:33:15.333787 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.333493 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" event={"ID":"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d","Type":"ContainerDied","Data":"a7130cb15d7212afe92be879b96e7a54013535f8b517cecb47f357f1f4b77442"} Oct 31 16:33:15.637115 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.637092 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 16:33:15.722527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.722486 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-openshift-service-ca\") pod \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " Oct 31 16:33:15.722746 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.722615 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-maas-api-tls\") pod \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " Oct 31 16:33:15.722746 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.722651 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-metrics-certs\") pod \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " Oct 31 16:33:15.722746 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.722685 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-kube-api-access-xqrgk\" (UniqueName: \"kubernetes.io/projected/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-kube-api-access-xqrgk\") pod \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\" (UID: \"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d\") " Oct 31 16:33:15.723050 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.723020 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-openshift-service-ca" pod "e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" (UID: "e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:33:15.725006 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.724970 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-metrics-certs" pod "e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" (UID: "e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:33:15.725140 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.725087 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-maas-api-tls" pod "e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" (UID: "e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:33:15.725140 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.725121 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-kube-api-access-xqrgk" pod "e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" (UID: "e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d"). InnerVolumeSpecName "kube-api-access-xqrgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:33:15.825388 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.824512 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqrgk\" (UniqueName: \"kubernetes.io/projected/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-kube-api-access-xqrgk\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:15.825388 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.824560 4399 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-openshift-service-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:15.825388 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.824577 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-maas-api-tls\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:15.825388 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:15.824593 4399 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d-metrics-certs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:16.351792 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:16.351741 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" event={"ID":"e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d","Type":"ContainerDied","Data":"57c881737358686a245b9c7358ea45251561d7fabe5439f3c7cf6db7fcfd74d8"} Oct 31 16:33:16.352472 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:16.352448 4399 scope.go:122] "RemoveContainer" containerID="a7130cb15d7212afe92be879b96e7a54013535f8b517cecb47f357f1f4b77442" Oct 31 16:33:16.352866 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:16.352849 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-84fff96864-g59d7" Oct 31 16:33:16.538027 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:16.537995 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-g59d7"] Oct 31 16:33:16.603674 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:16.603590 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-84fff96864-g59d7"] Oct 31 16:33:16.912817 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:16.912687 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" path="/var/lib/kubelet/pods/e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d/volumes" Oct 31 16:33:21.219618 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.219574 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2"] Oct 31 16:33:21.220218 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.220198 4399 cpu_manager.go:414] "containerMap: removing container" podUID="e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" containerName="maas-api" Oct 31 16:33:21.220324 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.220221 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" containerName="maas-api" Oct 31 16:33:21.220324 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.220243 4399 cpu_manager.go:414] "containerMap: removing container" podUID="5aac9441-7f91-49f3-9472-8fcd34fec615" containerName="payload-processing" Oct 31 16:33:21.220324 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.220251 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aac9441-7f91-49f3-9472-8fcd34fec615" containerName="payload-processing" Oct 31 16:33:21.220485 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.220325 4399 cpu_manager.go:414] "containerMap: removing container" podUID="8a05f09f-b51e-4f82-b934-7a697b365d80" containerName="payload-pre-processing" Oct 31 16:33:21.220485 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.220340 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a05f09f-b51e-4f82-b934-7a697b365d80" containerName="payload-pre-processing" Oct 31 16:33:21.220585 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.220552 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d8aad2-88c7-46fb-b730-b6aa7a2d4c2d" containerName="maas-api" Oct 31 16:33:21.220641 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.220581 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aac9441-7f91-49f3-9472-8fcd34fec615" containerName="payload-processing" Oct 31 16:33:21.220641 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.220613 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a05f09f-b51e-4f82-b934-7a697b365d80" containerName="payload-pre-processing" Oct 31 16:33:21.223829 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.223809 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.278868 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.278842 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2"] Oct 31 16:33:21.324743 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.324688 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:33:21.324743 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.324724 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:33:21.373042 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.373011 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhcn\" (UniqueName: \"kubernetes.io/projected/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-kube-api-access-rfhcn\") pod \"maas-api-revoke-sub-e75faa62-rphj2\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.373172 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.373071 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e75faa62-rphj2\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.373215 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.373188 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e75faa62-rphj2\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.474154 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.474083 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhcn\" (UniqueName: \"kubernetes.io/projected/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-kube-api-access-rfhcn\") pod \"maas-api-revoke-sub-e75faa62-rphj2\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.474154 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.474144 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e75faa62-rphj2\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.474325 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.474186 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e75faa62-rphj2\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.474971 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.474902 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e75faa62-rphj2\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.476732 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.476711 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e75faa62-rphj2\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.530038 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.530014 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhcn\" (UniqueName: \"kubernetes.io/projected/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-kube-api-access-rfhcn\") pod \"maas-api-revoke-sub-e75faa62-rphj2\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.536043 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.536021 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:21.908586 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:21.908555 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2"] Oct 31 16:33:21.910106 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:33:21.910075 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dd0f3f_be5d_4c6e_bd80_1f1ede469b3a.slice/crio-8fa2e409447da1dc07d0f35c93eff5abec04ed0d0f1850be88eba58c0a28281d WatchSource:0}: Error finding container 8fa2e409447da1dc07d0f35c93eff5abec04ed0d0f1850be88eba58c0a28281d: Status 404 returned error can't find the container with id 8fa2e409447da1dc07d0f35c93eff5abec04ed0d0f1850be88eba58c0a28281d Oct 31 16:33:22.384881 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:22.384850 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" event={"ID":"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a","Type":"ContainerStarted","Data":"9a13f6b8ad4f1708fea99b76c5c855fe83b5a24ed72daa4dbf489dd42b68f657"} Oct 31 16:33:22.385255 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:22.384903 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" event={"ID":"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a","Type":"ContainerStarted","Data":"8fa2e409447da1dc07d0f35c93eff5abec04ed0d0f1850be88eba58c0a28281d"} Oct 31 16:33:22.496190 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:22.496133 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" podStartSLOduration=1.496117514 podStartE2EDuration="1.496117514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:33:22.496077723 +0000 UTC m=+6445.707319131" watchObservedRunningTime="2026-10-31 16:33:22.496117514 +0000 UTC m=+6445.707358930" Oct 31 16:33:23.390968 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:23.390929 4399 generic.go:356] "Generic (PLEG): container finished" podID="42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a" containerID="9a13f6b8ad4f1708fea99b76c5c855fe83b5a24ed72daa4dbf489dd42b68f657" exitCode=0 Oct 31 16:33:23.391370 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:23.390988 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" event={"ID":"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a","Type":"ContainerDied","Data":"9a13f6b8ad4f1708fea99b76c5c855fe83b5a24ed72daa4dbf489dd42b68f657"} Oct 31 16:33:24.536422 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.536389 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:24.602079 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.602053 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-openshift-ca\") pod \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " Oct 31 16:33:24.602213 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.602084 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-xks-ca\") pod \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " Oct 31 16:33:24.602213 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.602152 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-kube-api-access-rfhcn\" (UniqueName: \"kubernetes.io/projected/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-kube-api-access-rfhcn\") pod \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\" (UID: \"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a\") " Oct 31 16:33:24.602369 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.602346 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-openshift-ca" pod "42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a" (UID: "42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:33:24.602464 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.602450 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:24.604205 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.604186 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-xks-ca" pod "42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a" (UID: "42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:33:24.604434 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.604413 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-kube-api-access-rfhcn" pod "42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a" (UID: "42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a"). InnerVolumeSpecName "kube-api-access-rfhcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:33:24.703434 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.703407 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfhcn\" (UniqueName: \"kubernetes.io/projected/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-kube-api-access-rfhcn\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:24.703434 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:24.703432 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:25.402868 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:25.402835 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" Oct 31 16:33:25.403055 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:25.402826 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2" event={"ID":"42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a","Type":"ContainerDied","Data":"8fa2e409447da1dc07d0f35c93eff5abec04ed0d0f1850be88eba58c0a28281d"} Oct 31 16:33:25.403055 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:25.402955 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa2e409447da1dc07d0f35c93eff5abec04ed0d0f1850be88eba58c0a28281d" Oct 31 16:33:26.205568 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:33:26.205533 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:33:31.979221 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:31.979190 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk"] Oct 31 16:33:31.979651 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:31.979635 4399 cpu_manager.go:414] "containerMap: removing container" podUID="42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a" containerName="revoke-keys" Oct 31 16:33:31.979711 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:31.979654 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a" containerName="revoke-keys" Oct 31 16:33:31.979780 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:31.979770 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a" containerName="revoke-keys" Oct 31 16:33:31.983044 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:31.983026 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.040084 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.040051 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk"] Oct 31 16:33:32.064138 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.064106 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc4542d-lhnjk\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.064329 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.064170 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffhfs\" (UniqueName: \"kubernetes.io/projected/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-kube-api-access-ffhfs\") pod \"maas-api-revoke-sub-0cc4542d-lhnjk\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.064329 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.064245 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc4542d-lhnjk\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.084285 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.084259 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:33:32.084418 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.084291 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:33:32.165058 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.165027 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc4542d-lhnjk\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.165197 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.165119 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc4542d-lhnjk\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.165197 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.165174 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffhfs\" (UniqueName: \"kubernetes.io/projected/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-kube-api-access-ffhfs\") pod \"maas-api-revoke-sub-0cc4542d-lhnjk\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.165750 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.165731 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cc4542d-lhnjk\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.167378 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.167362 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cc4542d-lhnjk\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.221050 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.221021 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffhfs\" (UniqueName: \"kubernetes.io/projected/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-kube-api-access-ffhfs\") pod \"maas-api-revoke-sub-0cc4542d-lhnjk\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.299009 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.298947 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:32.674176 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:32.674091 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk"] Oct 31 16:33:32.680334 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:33:32.680291 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5695b68a_b305_49d6_8bfd_5bd50e13a2d9.slice/crio-ec64a2661fa0510040802f2bf1ad897650739c1caa146cbb2944611c26be29ab WatchSource:0}: Error finding container ec64a2661fa0510040802f2bf1ad897650739c1caa146cbb2944611c26be29ab: Status 404 returned error can't find the container with id ec64a2661fa0510040802f2bf1ad897650739c1caa146cbb2944611c26be29ab Oct 31 16:33:33.440994 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:33.440896 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" event={"ID":"5695b68a-b305-49d6-8bfd-5bd50e13a2d9","Type":"ContainerStarted","Data":"58e7b951db52a25a1717cb06c42f07bfffb0acc5fb09e7311fa1f0eadeba14d4"} Oct 31 16:33:33.440994 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:33.440957 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" event={"ID":"5695b68a-b305-49d6-8bfd-5bd50e13a2d9","Type":"ContainerStarted","Data":"ec64a2661fa0510040802f2bf1ad897650739c1caa146cbb2944611c26be29ab"} Oct 31 16:33:33.555843 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:33.555796 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" podStartSLOduration=2.555782456 podStartE2EDuration="2.555782456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:33:33.552274713 +0000 UTC m=+6456.763516124" watchObservedRunningTime="2026-10-31 16:33:33.555782456 +0000 UTC m=+6456.767023891" Oct 31 16:33:34.447501 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:34.447460 4399 generic.go:356] "Generic (PLEG): container finished" podID="5695b68a-b305-49d6-8bfd-5bd50e13a2d9" containerID="58e7b951db52a25a1717cb06c42f07bfffb0acc5fb09e7311fa1f0eadeba14d4" exitCode=0 Oct 31 16:33:34.447926 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:34.447546 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" event={"ID":"5695b68a-b305-49d6-8bfd-5bd50e13a2d9","Type":"ContainerDied","Data":"58e7b951db52a25a1717cb06c42f07bfffb0acc5fb09e7311fa1f0eadeba14d4"} Oct 31 16:33:35.598080 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.598053 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:35.701582 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.701549 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-openshift-ca\") pod \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " Oct 31 16:33:35.701582 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.701586 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-xks-ca\") pod \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " Oct 31 16:33:35.701864 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.701766 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-kube-api-access-ffhfs\" (UniqueName: \"kubernetes.io/projected/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-kube-api-access-ffhfs\") pod \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\" (UID: \"5695b68a-b305-49d6-8bfd-5bd50e13a2d9\") " Oct 31 16:33:35.702048 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.702023 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-openshift-ca" pod "5695b68a-b305-49d6-8bfd-5bd50e13a2d9" (UID: "5695b68a-b305-49d6-8bfd-5bd50e13a2d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:33:35.704031 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.703966 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-xks-ca" pod "5695b68a-b305-49d6-8bfd-5bd50e13a2d9" (UID: "5695b68a-b305-49d6-8bfd-5bd50e13a2d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:33:35.704149 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.704035 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-kube-api-access-ffhfs" pod "5695b68a-b305-49d6-8bfd-5bd50e13a2d9" (UID: "5695b68a-b305-49d6-8bfd-5bd50e13a2d9"). InnerVolumeSpecName "kube-api-access-ffhfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:33:35.803019 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.802983 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffhfs\" (UniqueName: \"kubernetes.io/projected/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-kube-api-access-ffhfs\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:35.803019 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.803012 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:35.803019 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:35.803023 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5695b68a-b305-49d6-8bfd-5bd50e13a2d9-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:33:36.459105 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:36.459076 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" Oct 31 16:33:36.459288 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:36.459120 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk" event={"ID":"5695b68a-b305-49d6-8bfd-5bd50e13a2d9","Type":"ContainerDied","Data":"ec64a2661fa0510040802f2bf1ad897650739c1caa146cbb2944611c26be29ab"} Oct 31 16:33:36.459288 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:36.459165 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec64a2661fa0510040802f2bf1ad897650739c1caa146cbb2944611c26be29ab" Oct 31 16:33:54.296841 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:54.296801 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:33:54.297602 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:33:54.297568 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:33:56.207905 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:33:56.207865 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:34:05.469843 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.469807 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q"] Oct 31 16:34:05.470259 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.470245 4399 cpu_manager.go:414] "containerMap: removing container" podUID="5695b68a-b305-49d6-8bfd-5bd50e13a2d9" containerName="revoke-keys" Oct 31 16:34:05.470304 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.470260 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5695b68a-b305-49d6-8bfd-5bd50e13a2d9" containerName="revoke-keys" Oct 31 16:34:05.470349 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.470343 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="5695b68a-b305-49d6-8bfd-5bd50e13a2d9" containerName="revoke-keys" Oct 31 16:34:05.474718 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.474684 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.516461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.516433 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c086957d-4424q\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.516615 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.516482 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhx89\" (UniqueName: \"kubernetes.io/projected/08c42de4-2a0e-47e2-8c97-3473676e6e69-kube-api-access-zhx89\") pod \"maas-api-revoke-sub-c086957d-4424q\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.516615 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.516505 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c086957d-4424q\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.529941 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.529912 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q"] Oct 31 16:34:05.574984 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.574956 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:34:05.575119 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.575004 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:34:05.617112 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.617082 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c086957d-4424q\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.617244 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.617131 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhx89\" (UniqueName: \"kubernetes.io/projected/08c42de4-2a0e-47e2-8c97-3473676e6e69-kube-api-access-zhx89\") pod \"maas-api-revoke-sub-c086957d-4424q\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.617295 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.617243 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c086957d-4424q\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.617726 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.617683 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c086957d-4424q\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.619755 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.619727 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c086957d-4424q\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.672843 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.672820 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhx89\" (UniqueName: \"kubernetes.io/projected/08c42de4-2a0e-47e2-8c97-3473676e6e69-kube-api-access-zhx89\") pod \"maas-api-revoke-sub-c086957d-4424q\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:05.791940 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:05.791869 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:06.162975 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:06.162951 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q"] Oct 31 16:34:06.164711 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:34:06.164672 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c42de4_2a0e_47e2_8c97_3473676e6e69.slice/crio-62ba0c48d5cba85e507e9b3f31ea20797a0ec049f0509c28b8df75529c078c61 WatchSource:0}: Error finding container 62ba0c48d5cba85e507e9b3f31ea20797a0ec049f0509c28b8df75529c078c61: Status 404 returned error can't find the container with id 62ba0c48d5cba85e507e9b3f31ea20797a0ec049f0509c28b8df75529c078c61 Oct 31 16:34:06.645601 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:06.645563 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" event={"ID":"08c42de4-2a0e-47e2-8c97-3473676e6e69","Type":"ContainerStarted","Data":"4d2bb615c2f67ad6e711b27cef8865c96d9585a799b949bfde9e1aeeffa93d25"} Oct 31 16:34:06.646034 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:06.645623 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" event={"ID":"08c42de4-2a0e-47e2-8c97-3473676e6e69","Type":"ContainerStarted","Data":"62ba0c48d5cba85e507e9b3f31ea20797a0ec049f0509c28b8df75529c078c61"} Oct 31 16:34:06.761557 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:06.761448 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" podStartSLOduration=1.761430969 podStartE2EDuration="1.761430969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:34:06.759901999 +0000 UTC m=+6489.971143412" watchObservedRunningTime="2026-10-31 16:34:06.761430969 +0000 UTC m=+6489.972672381" Oct 31 16:34:07.651406 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:07.651373 4399 generic.go:356] "Generic (PLEG): container finished" podID="08c42de4-2a0e-47e2-8c97-3473676e6e69" containerID="4d2bb615c2f67ad6e711b27cef8865c96d9585a799b949bfde9e1aeeffa93d25" exitCode=0 Oct 31 16:34:07.651819 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:07.651459 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" event={"ID":"08c42de4-2a0e-47e2-8c97-3473676e6e69","Type":"ContainerDied","Data":"4d2bb615c2f67ad6e711b27cef8865c96d9585a799b949bfde9e1aeeffa93d25"} Oct 31 16:34:08.796936 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.796908 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:08.848645 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.848609 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c42de4-2a0e-47e2-8c97-3473676e6e69-kube-api-access-zhx89\" (UniqueName: \"kubernetes.io/projected/08c42de4-2a0e-47e2-8c97-3473676e6e69-kube-api-access-zhx89\") pod \"08c42de4-2a0e-47e2-8c97-3473676e6e69\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " Oct 31 16:34:08.848842 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.848818 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-openshift-ca\") pod \"08c42de4-2a0e-47e2-8c97-3473676e6e69\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " Oct 31 16:34:08.848920 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.848882 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-xks-ca\") pod \"08c42de4-2a0e-47e2-8c97-3473676e6e69\" (UID: \"08c42de4-2a0e-47e2-8c97-3473676e6e69\") " Oct 31 16:34:08.849188 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.849160 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-openshift-ca" pod "08c42de4-2a0e-47e2-8c97-3473676e6e69" (UID: "08c42de4-2a0e-47e2-8c97-3473676e6e69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:34:08.850832 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.850809 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c42de4-2a0e-47e2-8c97-3473676e6e69-kube-api-access-zhx89" pod "08c42de4-2a0e-47e2-8c97-3473676e6e69" (UID: "08c42de4-2a0e-47e2-8c97-3473676e6e69"). InnerVolumeSpecName "kube-api-access-zhx89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:08.850930 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.850862 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-xks-ca" pod "08c42de4-2a0e-47e2-8c97-3473676e6e69" (UID: "08c42de4-2a0e-47e2-8c97-3473676e6e69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:34:08.950345 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.950321 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhx89\" (UniqueName: \"kubernetes.io/projected/08c42de4-2a0e-47e2-8c97-3473676e6e69-kube-api-access-zhx89\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:34:08.950496 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.950353 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:34:08.950496 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:08.950364 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c42de4-2a0e-47e2-8c97-3473676e6e69-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:34:09.662611 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:09.662579 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" Oct 31 16:34:09.662819 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:09.662605 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q" event={"ID":"08c42de4-2a0e-47e2-8c97-3473676e6e69","Type":"ContainerDied","Data":"62ba0c48d5cba85e507e9b3f31ea20797a0ec049f0509c28b8df75529c078c61"} Oct 31 16:34:09.662819 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:09.662662 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62ba0c48d5cba85e507e9b3f31ea20797a0ec049f0509c28b8df75529c078c61" Oct 31 16:34:26.238888 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:34:26.238846 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:34:48.160375 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.160283 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r"] Oct 31 16:34:48.160853 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.160839 4399 cpu_manager.go:414] "containerMap: removing container" podUID="08c42de4-2a0e-47e2-8c97-3473676e6e69" containerName="revoke-keys" Oct 31 16:34:48.160906 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.160854 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c42de4-2a0e-47e2-8c97-3473676e6e69" containerName="revoke-keys" Oct 31 16:34:48.160978 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.160943 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c42de4-2a0e-47e2-8c97-3473676e6e69" containerName="revoke-keys" Oct 31 16:34:48.164467 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.164449 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.219925 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.219887 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r"] Oct 31 16:34:48.265084 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.265047 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:34:48.265275 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.265105 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:34:48.275060 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.275025 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5h65\" (UniqueName: \"kubernetes.io/projected/89613097-30b1-4a69-86f8-55f7312f5c5d-kube-api-access-l5h65\") pod \"maas-api-revoke-sub-1a0d4800-2442r\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.275286 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.275253 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a0d4800-2442r\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.275410 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.275310 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a0d4800-2442r\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.376768 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.376688 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a0d4800-2442r\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.376996 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.376780 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a0d4800-2442r\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.376996 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.376847 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5h65\" (UniqueName: \"kubernetes.io/projected/89613097-30b1-4a69-86f8-55f7312f5c5d-kube-api-access-l5h65\") pod \"maas-api-revoke-sub-1a0d4800-2442r\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.377408 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.377384 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1a0d4800-2442r\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.379394 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.379370 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1a0d4800-2442r\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.433100 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.433017 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5h65\" (UniqueName: \"kubernetes.io/projected/89613097-30b1-4a69-86f8-55f7312f5c5d-kube-api-access-l5h65\") pod \"maas-api-revoke-sub-1a0d4800-2442r\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.477878 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.477848 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:48.850234 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.850204 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r"] Oct 31 16:34:48.851962 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:34:48.851935 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89613097_30b1_4a69_86f8_55f7312f5c5d.slice/crio-b9eacf501786dad5e3abd16e5233db4773cf9780f1c3e433dd2945d29ed88b80 WatchSource:0}: Error finding container b9eacf501786dad5e3abd16e5233db4773cf9780f1c3e433dd2945d29ed88b80: Status 404 returned error can't find the container with id b9eacf501786dad5e3abd16e5233db4773cf9780f1c3e433dd2945d29ed88b80 Oct 31 16:34:48.895146 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:48.895118 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" event={"ID":"89613097-30b1-4a69-86f8-55f7312f5c5d","Type":"ContainerStarted","Data":"b9eacf501786dad5e3abd16e5233db4773cf9780f1c3e433dd2945d29ed88b80"} Oct 31 16:34:49.897942 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:49.897907 4399 generic.go:356] "Generic (PLEG): container finished" podID="89613097-30b1-4a69-86f8-55f7312f5c5d" containerID="383cfe5d0de3efc3aa71ebfaf5ab75ee83a1635da2040ad91ca1c0181d79f371" exitCode=0 Oct 31 16:34:49.898324 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:49.897998 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" event={"ID":"89613097-30b1-4a69-86f8-55f7312f5c5d","Type":"ContainerDied","Data":"383cfe5d0de3efc3aa71ebfaf5ab75ee83a1635da2040ad91ca1c0181d79f371"} Oct 31 16:34:51.046560 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.046536 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:51.201940 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.201914 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89613097-30b1-4a69-86f8-55f7312f5c5d-kube-api-access-l5h65\" (UniqueName: \"kubernetes.io/projected/89613097-30b1-4a69-86f8-55f7312f5c5d-kube-api-access-l5h65\") pod \"89613097-30b1-4a69-86f8-55f7312f5c5d\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " Oct 31 16:34:51.202094 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.201966 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-xks-ca\") pod \"89613097-30b1-4a69-86f8-55f7312f5c5d\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " Oct 31 16:34:51.202094 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.201994 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-openshift-ca\") pod \"89613097-30b1-4a69-86f8-55f7312f5c5d\" (UID: \"89613097-30b1-4a69-86f8-55f7312f5c5d\") " Oct 31 16:34:51.202441 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.202417 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-openshift-ca" pod "89613097-30b1-4a69-86f8-55f7312f5c5d" (UID: "89613097-30b1-4a69-86f8-55f7312f5c5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:34:51.204111 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.204081 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-xks-ca" pod "89613097-30b1-4a69-86f8-55f7312f5c5d" (UID: "89613097-30b1-4a69-86f8-55f7312f5c5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:34:51.204222 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.204177 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89613097-30b1-4a69-86f8-55f7312f5c5d-kube-api-access-l5h65" pod "89613097-30b1-4a69-86f8-55f7312f5c5d" (UID: "89613097-30b1-4a69-86f8-55f7312f5c5d"). InnerVolumeSpecName "kube-api-access-l5h65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:34:51.303618 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.303591 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5h65\" (UniqueName: \"kubernetes.io/projected/89613097-30b1-4a69-86f8-55f7312f5c5d-kube-api-access-l5h65\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:34:51.303618 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.303616 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:34:51.303786 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.303625 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89613097-30b1-4a69-86f8-55f7312f5c5d-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:34:51.908837 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.908805 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" Oct 31 16:34:51.909020 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.908847 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r" event={"ID":"89613097-30b1-4a69-86f8-55f7312f5c5d","Type":"ContainerDied","Data":"b9eacf501786dad5e3abd16e5233db4773cf9780f1c3e433dd2945d29ed88b80"} Oct 31 16:34:51.909020 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:51.908887 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9eacf501786dad5e3abd16e5233db4773cf9780f1c3e433dd2945d29ed88b80" Oct 31 16:34:56.207121 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:34:56.207080 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:34:58.968102 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:58.968067 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn"] Oct 31 16:34:58.968735 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:58.968720 4399 cpu_manager.go:414] "containerMap: removing container" podUID="89613097-30b1-4a69-86f8-55f7312f5c5d" containerName="revoke-keys" Oct 31 16:34:58.968796 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:58.968738 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89613097-30b1-4a69-86f8-55f7312f5c5d" containerName="revoke-keys" Oct 31 16:34:58.968851 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:58.968840 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="89613097-30b1-4a69-86f8-55f7312f5c5d" containerName="revoke-keys" Oct 31 16:34:58.972464 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:58.972446 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.026343 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.026314 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn"] Oct 31 16:34:59.071596 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.071562 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:34:59.071596 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.071574 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:34:59.078924 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.078894 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1efce29-wkrnn\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.079115 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.079092 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtw6p\" (UniqueName: \"kubernetes.io/projected/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-kube-api-access-xtw6p\") pod \"maas-api-revoke-sub-f1efce29-wkrnn\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.079273 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.079250 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1efce29-wkrnn\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.180630 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.180583 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtw6p\" (UniqueName: \"kubernetes.io/projected/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-kube-api-access-xtw6p\") pod \"maas-api-revoke-sub-f1efce29-wkrnn\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.180830 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.180734 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1efce29-wkrnn\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.180830 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.180775 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1efce29-wkrnn\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.181360 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.181338 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f1efce29-wkrnn\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.183281 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.183255 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f1efce29-wkrnn\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.236981 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.236910 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtw6p\" (UniqueName: \"kubernetes.io/projected/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-kube-api-access-xtw6p\") pod \"maas-api-revoke-sub-f1efce29-wkrnn\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.285983 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.285949 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:34:59.659251 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.659216 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn"] Oct 31 16:34:59.660611 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:34:59.660583 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed9cc4f_cffb_4774_b37b_cf729e8ebaa6.slice/crio-514d4a1c17e638a0155be2cd60c9e64e7e74b75683e984d0e46ebff1c9595eb2 WatchSource:0}: Error finding container 514d4a1c17e638a0155be2cd60c9e64e7e74b75683e984d0e46ebff1c9595eb2: Status 404 returned error can't find the container with id 514d4a1c17e638a0155be2cd60c9e64e7e74b75683e984d0e46ebff1c9595eb2 Oct 31 16:34:59.948661 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:34:59.948629 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" event={"ID":"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6","Type":"ContainerStarted","Data":"514d4a1c17e638a0155be2cd60c9e64e7e74b75683e984d0e46ebff1c9595eb2"} Oct 31 16:35:00.954614 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:00.954576 4399 generic.go:356] "Generic (PLEG): container finished" podID="bed9cc4f-cffb-4774-b37b-cf729e8ebaa6" containerID="34d8e77dff1f95b59b8603abaf743eaf7a5bbf7ad89188d2226756dd93cd5b03" exitCode=0 Oct 31 16:35:00.955023 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:00.954654 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" event={"ID":"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6","Type":"ContainerDied","Data":"34d8e77dff1f95b59b8603abaf743eaf7a5bbf7ad89188d2226756dd93cd5b03"} Oct 31 16:35:02.101900 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.101878 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:35:02.213646 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.213555 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-openshift-ca\") pod \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " Oct 31 16:35:02.213872 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.213657 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-xks-ca\") pod \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " Oct 31 16:35:02.214051 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.214022 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-openshift-ca" pod "bed9cc4f-cffb-4774-b37b-cf729e8ebaa6" (UID: "bed9cc4f-cffb-4774-b37b-cf729e8ebaa6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:35:02.214497 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.214469 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-kube-api-access-xtw6p\" (UniqueName: \"kubernetes.io/projected/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-kube-api-access-xtw6p\") pod \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\" (UID: \"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6\") " Oct 31 16:35:02.214912 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.214886 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.216445 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.216421 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-xks-ca" pod "bed9cc4f-cffb-4774-b37b-cf729e8ebaa6" (UID: "bed9cc4f-cffb-4774-b37b-cf729e8ebaa6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:35:02.217093 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.217058 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-kube-api-access-xtw6p" pod "bed9cc4f-cffb-4774-b37b-cf729e8ebaa6" (UID: "bed9cc4f-cffb-4774-b37b-cf729e8ebaa6"). InnerVolumeSpecName "kube-api-access-xtw6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:02.315949 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.315919 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.315949 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.315948 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtw6p\" (UniqueName: \"kubernetes.io/projected/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6-kube-api-access-xtw6p\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:35:02.965004 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.964978 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" Oct 31 16:35:02.965165 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.965014 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn" event={"ID":"bed9cc4f-cffb-4774-b37b-cf729e8ebaa6","Type":"ContainerDied","Data":"514d4a1c17e638a0155be2cd60c9e64e7e74b75683e984d0e46ebff1c9595eb2"} Oct 31 16:35:02.965165 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:02.965053 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="514d4a1c17e638a0155be2cd60c9e64e7e74b75683e984d0e46ebff1c9595eb2" Oct 31 16:35:26.237009 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:35:26.236970 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:35:39.299848 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:39.299806 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xgc5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:35:39.300929 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:39.300893 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xgc5z" podUID="116d08e1-3b25-43fb-92e9-2b960cd2e50d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:35:47.163087 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.163051 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862"] Oct 31 16:35:47.163575 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.163464 4399 cpu_manager.go:414] "containerMap: removing container" podUID="bed9cc4f-cffb-4774-b37b-cf729e8ebaa6" containerName="revoke-keys" Oct 31 16:35:47.163575 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.163475 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bed9cc4f-cffb-4774-b37b-cf729e8ebaa6" containerName="revoke-keys" Oct 31 16:35:47.163575 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.163565 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="bed9cc4f-cffb-4774-b37b-cf729e8ebaa6" containerName="revoke-keys" Oct 31 16:35:47.166599 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.166580 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.213662 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.213641 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vb4p\" (UniqueName: \"kubernetes.io/projected/6559c121-9218-4b26-a1c3-684bb352e257-kube-api-access-7vb4p\") pod \"maas-api-revoke-sub-502a5a0a-v8862\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.213780 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.213672 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-502a5a0a-v8862\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.213834 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.213813 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-502a5a0a-v8862\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.223669 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.223645 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862"] Oct 31 16:35:47.267548 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.267524 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:35:47.267748 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.267524 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:35:47.314331 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.314306 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vb4p\" (UniqueName: \"kubernetes.io/projected/6559c121-9218-4b26-a1c3-684bb352e257-kube-api-access-7vb4p\") pod \"maas-api-revoke-sub-502a5a0a-v8862\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.314485 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.314350 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-502a5a0a-v8862\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.314485 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.314448 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-502a5a0a-v8862\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.315384 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.315350 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-502a5a0a-v8862\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.317193 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.317173 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-xks-ca\") pod \"maas-api-revoke-sub-502a5a0a-v8862\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.370511 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.370483 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vb4p\" (UniqueName: \"kubernetes.io/projected/6559c121-9218-4b26-a1c3-684bb352e257-kube-api-access-7vb4p\") pod \"maas-api-revoke-sub-502a5a0a-v8862\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.479722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.479685 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:47.849995 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:47.849964 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862"] Oct 31 16:35:47.851231 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:35:47.851208 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6559c121_9218_4b26_a1c3_684bb352e257.slice/crio-bcf81890939579814a06de56249f2a92196b231bf48c98e59d110fd4f97e7ff8 WatchSource:0}: Error finding container bcf81890939579814a06de56249f2a92196b231bf48c98e59d110fd4f97e7ff8: Status 404 returned error can't find the container with id bcf81890939579814a06de56249f2a92196b231bf48c98e59d110fd4f97e7ff8 Oct 31 16:35:48.263689 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:48.263651 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" event={"ID":"6559c121-9218-4b26-a1c3-684bb352e257","Type":"ContainerStarted","Data":"4bcc121c72c48a4cdf144d50b64253cce43b6440a79e4ba9feeebdda1847fba9"} Oct 31 16:35:48.264023 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:48.263780 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" event={"ID":"6559c121-9218-4b26-a1c3-684bb352e257","Type":"ContainerStarted","Data":"bcf81890939579814a06de56249f2a92196b231bf48c98e59d110fd4f97e7ff8"} Oct 31 16:35:48.376170 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:48.376120 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" podStartSLOduration=1.376105385 podStartE2EDuration="1.376105385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:35:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:48.374675908 +0000 UTC m=+6591.585917348" watchObservedRunningTime="2026-10-31 16:35:48.376105385 +0000 UTC m=+6591.587346797" Oct 31 16:35:49.270195 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:49.270157 4399 generic.go:356] "Generic (PLEG): container finished" podID="6559c121-9218-4b26-a1c3-684bb352e257" containerID="4bcc121c72c48a4cdf144d50b64253cce43b6440a79e4ba9feeebdda1847fba9" exitCode=0 Oct 31 16:35:49.270618 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:49.270242 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" event={"ID":"6559c121-9218-4b26-a1c3-684bb352e257","Type":"ContainerDied","Data":"4bcc121c72c48a4cdf144d50b64253cce43b6440a79e4ba9feeebdda1847fba9"} Oct 31 16:35:50.412260 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.412230 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:50.546973 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.546883 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-xks-ca\") pod \"6559c121-9218-4b26-a1c3-684bb352e257\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " Oct 31 16:35:50.546973 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.546935 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6559c121-9218-4b26-a1c3-684bb352e257-kube-api-access-7vb4p\" (UniqueName: \"kubernetes.io/projected/6559c121-9218-4b26-a1c3-684bb352e257-kube-api-access-7vb4p\") pod \"6559c121-9218-4b26-a1c3-684bb352e257\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " Oct 31 16:35:50.547221 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.547066 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-openshift-ca\") pod \"6559c121-9218-4b26-a1c3-684bb352e257\" (UID: \"6559c121-9218-4b26-a1c3-684bb352e257\") " Oct 31 16:35:50.547440 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.547414 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-openshift-ca" pod "6559c121-9218-4b26-a1c3-684bb352e257" (UID: "6559c121-9218-4b26-a1c3-684bb352e257"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:35:50.549527 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.549496 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6559c121-9218-4b26-a1c3-684bb352e257-kube-api-access-7vb4p" pod "6559c121-9218-4b26-a1c3-684bb352e257" (UID: "6559c121-9218-4b26-a1c3-684bb352e257"). InnerVolumeSpecName "kube-api-access-7vb4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:35:50.549675 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.549634 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-xks-ca" pod "6559c121-9218-4b26-a1c3-684bb352e257" (UID: "6559c121-9218-4b26-a1c3-684bb352e257"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:35:50.648678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.648637 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:35:50.648678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.648674 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6559c121-9218-4b26-a1c3-684bb352e257-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:35:50.648678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:50.648684 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vb4p\" (UniqueName: \"kubernetes.io/projected/6559c121-9218-4b26-a1c3-684bb352e257-kube-api-access-7vb4p\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:35:51.280804 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:51.280776 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" Oct 31 16:35:51.280994 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:51.280775 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502a5a0a-v8862" event={"ID":"6559c121-9218-4b26-a1c3-684bb352e257","Type":"ContainerDied","Data":"bcf81890939579814a06de56249f2a92196b231bf48c98e59d110fd4f97e7ff8"} Oct 31 16:35:51.280994 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:51.280902 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf81890939579814a06de56249f2a92196b231bf48c98e59d110fd4f97e7ff8" Oct 31 16:35:56.206303 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:35:56.206266 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:35:57.915931 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:57.915899 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9"] Oct 31 16:35:57.916323 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:57.916307 4399 cpu_manager.go:414] "containerMap: removing container" podUID="6559c121-9218-4b26-a1c3-684bb352e257" containerName="revoke-keys" Oct 31 16:35:57.916368 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:57.916326 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6559c121-9218-4b26-a1c3-684bb352e257" containerName="revoke-keys" Oct 31 16:35:57.916454 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:57.916444 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="6559c121-9218-4b26-a1c3-684bb352e257" containerName="revoke-keys" Oct 31 16:35:57.919751 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:57.919729 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:57.976275 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:57.976248 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9"] Oct 31 16:35:58.019151 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.019112 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29v44\" (UniqueName: \"kubernetes.io/projected/39528537-4130-47e3-88e7-4b5c6bdf6e66-kube-api-access-29v44\") pod \"maas-api-revoke-sub-e62685e0-lt4q9\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.019309 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.019260 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e62685e0-lt4q9\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.019381 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.019362 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e62685e0-lt4q9\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.020242 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.020222 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:35:58.020364 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.020334 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:35:58.120306 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.120282 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e62685e0-lt4q9\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.120465 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.120327 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e62685e0-lt4q9\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.120465 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.120375 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29v44\" (UniqueName: \"kubernetes.io/projected/39528537-4130-47e3-88e7-4b5c6bdf6e66-kube-api-access-29v44\") pod \"maas-api-revoke-sub-e62685e0-lt4q9\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.120997 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.120975 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e62685e0-lt4q9\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.123016 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.122994 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e62685e0-lt4q9\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.176667 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.176612 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29v44\" (UniqueName: \"kubernetes.io/projected/39528537-4130-47e3-88e7-4b5c6bdf6e66-kube-api-access-29v44\") pod \"maas-api-revoke-sub-e62685e0-lt4q9\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.232586 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.232566 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:35:58.608300 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:58.608274 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9"] Oct 31 16:35:58.609665 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:35:58.609617 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39528537_4130_47e3_88e7_4b5c6bdf6e66.slice/crio-7f69a2b049da8ddf6a22625195891c8ed13b33a9f214c3df9907ab024b34bf62 WatchSource:0}: Error finding container 7f69a2b049da8ddf6a22625195891c8ed13b33a9f214c3df9907ab024b34bf62: Status 404 returned error can't find the container with id 7f69a2b049da8ddf6a22625195891c8ed13b33a9f214c3df9907ab024b34bf62 Oct 31 16:35:59.322770 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:59.322729 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" event={"ID":"39528537-4130-47e3-88e7-4b5c6bdf6e66","Type":"ContainerStarted","Data":"0833b2850875c657f9576a7be05b88f52af535d0e3bf3ccbafd5986a825f779d"} Oct 31 16:35:59.323197 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:59.322796 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" event={"ID":"39528537-4130-47e3-88e7-4b5c6bdf6e66","Type":"ContainerStarted","Data":"7f69a2b049da8ddf6a22625195891c8ed13b33a9f214c3df9907ab024b34bf62"} Oct 31 16:35:59.435876 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:35:59.435821 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" podStartSLOduration=2.435806862 podStartE2EDuration="2.435806862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:35:59.434234628 +0000 UTC m=+6602.645476040" watchObservedRunningTime="2026-10-31 16:35:59.435806862 +0000 UTC m=+6602.647048274" Oct 31 16:36:00.329870 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:00.329837 4399 generic.go:356] "Generic (PLEG): container finished" podID="39528537-4130-47e3-88e7-4b5c6bdf6e66" containerID="0833b2850875c657f9576a7be05b88f52af535d0e3bf3ccbafd5986a825f779d" exitCode=0 Oct 31 16:36:00.330295 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:00.329901 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" event={"ID":"39528537-4130-47e3-88e7-4b5c6bdf6e66","Type":"ContainerDied","Data":"0833b2850875c657f9576a7be05b88f52af535d0e3bf3ccbafd5986a825f779d"} Oct 31 16:36:01.499730 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.499675 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:36:01.658781 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.658057 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-openshift-ca\") pod \"39528537-4130-47e3-88e7-4b5c6bdf6e66\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " Oct 31 16:36:01.658781 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.658140 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39528537-4130-47e3-88e7-4b5c6bdf6e66-kube-api-access-29v44\" (UniqueName: \"kubernetes.io/projected/39528537-4130-47e3-88e7-4b5c6bdf6e66-kube-api-access-29v44\") pod \"39528537-4130-47e3-88e7-4b5c6bdf6e66\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " Oct 31 16:36:01.658781 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.658262 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-xks-ca\") pod \"39528537-4130-47e3-88e7-4b5c6bdf6e66\" (UID: \"39528537-4130-47e3-88e7-4b5c6bdf6e66\") " Oct 31 16:36:01.671971 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.671939 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-openshift-ca" pod "39528537-4130-47e3-88e7-4b5c6bdf6e66" (UID: "39528537-4130-47e3-88e7-4b5c6bdf6e66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:36:01.703890 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.703841 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-xks-ca" pod "39528537-4130-47e3-88e7-4b5c6bdf6e66" (UID: "39528537-4130-47e3-88e7-4b5c6bdf6e66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:36:01.717586 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.716946 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39528537-4130-47e3-88e7-4b5c6bdf6e66-kube-api-access-29v44" pod "39528537-4130-47e3-88e7-4b5c6bdf6e66" (UID: "39528537-4130-47e3-88e7-4b5c6bdf6e66"). InnerVolumeSpecName "kube-api-access-29v44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:36:01.759450 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.759343 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:36:01.759450 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.759396 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39528537-4130-47e3-88e7-4b5c6bdf6e66-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:36:01.759450 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:01.759414 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29v44\" (UniqueName: \"kubernetes.io/projected/39528537-4130-47e3-88e7-4b5c6bdf6e66-kube-api-access-29v44\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:36:02.353559 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:02.353521 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" event={"ID":"39528537-4130-47e3-88e7-4b5c6bdf6e66","Type":"ContainerDied","Data":"7f69a2b049da8ddf6a22625195891c8ed13b33a9f214c3df9907ab024b34bf62"} Oct 31 16:36:02.353906 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:02.353877 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f69a2b049da8ddf6a22625195891c8ed13b33a9f214c3df9907ab024b34bf62" Oct 31 16:36:02.354143 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:02.354124 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e62685e0-lt4q9" Oct 31 16:36:26.205722 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:36:26.205656 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:36:46.389099 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.389064 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p"] Oct 31 16:36:46.389525 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.389511 4399 cpu_manager.go:414] "containerMap: removing container" podUID="39528537-4130-47e3-88e7-4b5c6bdf6e66" containerName="revoke-keys" Oct 31 16:36:46.389570 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.389526 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39528537-4130-47e3-88e7-4b5c6bdf6e66" containerName="revoke-keys" Oct 31 16:36:46.389605 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.389600 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="39528537-4130-47e3-88e7-4b5c6bdf6e66" containerName="revoke-keys" Oct 31 16:36:46.392949 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.392933 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.446082 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.446011 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:36:46.450035 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.450009 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p"] Oct 31 16:36:46.455482 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.455461 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c8c7538-4pc9p\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.455610 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.455509 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c8c7538-4pc9p\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.455765 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.455741 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgghf\" (UniqueName: \"kubernetes.io/projected/cddb7bbc-ab64-479d-87db-93837af0bff7-kube-api-access-jgghf\") pod \"maas-api-revoke-sub-0c8c7538-4pc9p\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.495169 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.495147 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:36:46.556299 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.556265 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgghf\" (UniqueName: \"kubernetes.io/projected/cddb7bbc-ab64-479d-87db-93837af0bff7-kube-api-access-jgghf\") pod \"maas-api-revoke-sub-0c8c7538-4pc9p\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.556458 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.556337 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c8c7538-4pc9p\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.556458 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.556410 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c8c7538-4pc9p\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.557261 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.557231 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c8c7538-4pc9p\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.558893 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.558872 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c8c7538-4pc9p\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.612255 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.612228 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgghf\" (UniqueName: \"kubernetes.io/projected/cddb7bbc-ab64-479d-87db-93837af0bff7-kube-api-access-jgghf\") pod \"maas-api-revoke-sub-0c8c7538-4pc9p\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:46.705990 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:46.705961 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:47.080010 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:47.079984 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p"] Oct 31 16:36:47.083122 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:36:47.083081 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddb7bbc_ab64_479d_87db_93837af0bff7.slice/crio-10e6c807bf164b8e1c0c7adfff35d6f6aeb02d81d6b2cac2cd54b9f0096a8498 WatchSource:0}: Error finding container 10e6c807bf164b8e1c0c7adfff35d6f6aeb02d81d6b2cac2cd54b9f0096a8498: Status 404 returned error can't find the container with id 10e6c807bf164b8e1c0c7adfff35d6f6aeb02d81d6b2cac2cd54b9f0096a8498 Oct 31 16:36:47.720321 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:47.720286 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" event={"ID":"cddb7bbc-ab64-479d-87db-93837af0bff7","Type":"ContainerStarted","Data":"15caeb50083ba6112fc1725fcb993489b0afe026adf017c3648830e11bb31948"} Oct 31 16:36:47.720740 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:47.720340 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" event={"ID":"cddb7bbc-ab64-479d-87db-93837af0bff7","Type":"ContainerStarted","Data":"10e6c807bf164b8e1c0c7adfff35d6f6aeb02d81d6b2cac2cd54b9f0096a8498"} Oct 31 16:36:47.834111 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:47.834015 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" podStartSLOduration=1.833997665 podStartE2EDuration="1.833997665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:36:47.833488305 +0000 UTC m=+6651.044729716" watchObservedRunningTime="2026-10-31 16:36:47.833997665 +0000 UTC m=+6651.045239080" Oct 31 16:36:48.726368 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:48.726337 4399 generic.go:356] "Generic (PLEG): container finished" podID="cddb7bbc-ab64-479d-87db-93837af0bff7" containerID="15caeb50083ba6112fc1725fcb993489b0afe026adf017c3648830e11bb31948" exitCode=0 Oct 31 16:36:48.726786 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:48.726437 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" event={"ID":"cddb7bbc-ab64-479d-87db-93837af0bff7","Type":"ContainerDied","Data":"15caeb50083ba6112fc1725fcb993489b0afe026adf017c3648830e11bb31948"} Oct 31 16:36:49.868271 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:49.868246 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:49.987632 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:49.987536 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-openshift-ca\") pod \"cddb7bbc-ab64-479d-87db-93837af0bff7\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " Oct 31 16:36:49.987821 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:49.987654 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cddb7bbc-ab64-479d-87db-93837af0bff7-kube-api-access-jgghf\" (UniqueName: \"kubernetes.io/projected/cddb7bbc-ab64-479d-87db-93837af0bff7-kube-api-access-jgghf\") pod \"cddb7bbc-ab64-479d-87db-93837af0bff7\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " Oct 31 16:36:49.987821 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:49.987693 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-xks-ca\") pod \"cddb7bbc-ab64-479d-87db-93837af0bff7\" (UID: \"cddb7bbc-ab64-479d-87db-93837af0bff7\") " Oct 31 16:36:49.988024 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:49.987997 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-openshift-ca" pod "cddb7bbc-ab64-479d-87db-93837af0bff7" (UID: "cddb7bbc-ab64-479d-87db-93837af0bff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:36:49.988162 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:49.988143 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:36:49.990278 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:49.990255 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cddb7bbc-ab64-479d-87db-93837af0bff7-kube-api-access-jgghf" pod "cddb7bbc-ab64-479d-87db-93837af0bff7" (UID: "cddb7bbc-ab64-479d-87db-93837af0bff7"). InnerVolumeSpecName "kube-api-access-jgghf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:36:49.990624 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:49.990609 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-xks-ca" pod "cddb7bbc-ab64-479d-87db-93837af0bff7" (UID: "cddb7bbc-ab64-479d-87db-93837af0bff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:36:50.089604 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:50.089576 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgghf\" (UniqueName: \"kubernetes.io/projected/cddb7bbc-ab64-479d-87db-93837af0bff7-kube-api-access-jgghf\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:36:50.089604 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:50.089604 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cddb7bbc-ab64-479d-87db-93837af0bff7-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:36:50.739345 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:50.739313 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" Oct 31 16:36:50.739554 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:50.739351 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c8c7538-4pc9p" event={"ID":"cddb7bbc-ab64-479d-87db-93837af0bff7","Type":"ContainerDied","Data":"10e6c807bf164b8e1c0c7adfff35d6f6aeb02d81d6b2cac2cd54b9f0096a8498"} Oct 31 16:36:50.739554 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:50.739397 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10e6c807bf164b8e1c0c7adfff35d6f6aeb02d81d6b2cac2cd54b9f0096a8498" Oct 31 16:36:56.206469 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:36:56.206435 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:36:57.147077 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.147048 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9"] Oct 31 16:36:57.147461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.147447 4399 cpu_manager.go:414] "containerMap: removing container" podUID="cddb7bbc-ab64-479d-87db-93837af0bff7" containerName="revoke-keys" Oct 31 16:36:57.147461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.147461 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cddb7bbc-ab64-479d-87db-93837af0bff7" containerName="revoke-keys" Oct 31 16:36:57.147597 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.147572 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="cddb7bbc-ab64-479d-87db-93837af0bff7" containerName="revoke-keys" Oct 31 16:36:57.150571 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.150547 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.207965 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.207932 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9"] Oct 31 16:36:57.251678 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.251644 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:36:57.251852 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.251724 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:36:57.257725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.257680 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnd77\" (UniqueName: \"kubernetes.io/projected/82157379-54a9-461f-aa1a-f6e443df2d2d-kube-api-access-lnd77\") pod \"maas-api-revoke-sub-d1431612-dxjx9\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.257817 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.257802 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1431612-dxjx9\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.257872 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.257861 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1431612-dxjx9\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.358666 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.358633 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnd77\" (UniqueName: \"kubernetes.io/projected/82157379-54a9-461f-aa1a-f6e443df2d2d-kube-api-access-lnd77\") pod \"maas-api-revoke-sub-d1431612-dxjx9\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.358869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.358727 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1431612-dxjx9\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.358869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.358794 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1431612-dxjx9\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.359334 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.359316 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1431612-dxjx9\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.361216 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.361197 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1431612-dxjx9\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.414732 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.414651 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnd77\" (UniqueName: \"kubernetes.io/projected/82157379-54a9-461f-aa1a-f6e443df2d2d-kube-api-access-lnd77\") pod \"maas-api-revoke-sub-d1431612-dxjx9\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.463266 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.463242 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:36:57.836642 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:57.836613 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9"] Oct 31 16:36:57.837407 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:36:57.837377 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82157379_54a9_461f_aa1a_f6e443df2d2d.slice/crio-59438f0678f90554b9d553e622a5d2353b6ab42fd96819377b1a82c919dd79a7 WatchSource:0}: Error finding container 59438f0678f90554b9d553e622a5d2353b6ab42fd96819377b1a82c919dd79a7: Status 404 returned error can't find the container with id 59438f0678f90554b9d553e622a5d2353b6ab42fd96819377b1a82c919dd79a7 Oct 31 16:36:58.779080 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:58.779051 4399 generic.go:356] "Generic (PLEG): container finished" podID="82157379-54a9-461f-aa1a-f6e443df2d2d" containerID="a4840d3dc525ff27ca5bf75d075a07d82399b8cf74b2039bf09caee449405a59" exitCode=0 Oct 31 16:36:58.779429 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:58.779141 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" event={"ID":"82157379-54a9-461f-aa1a-f6e443df2d2d","Type":"ContainerDied","Data":"a4840d3dc525ff27ca5bf75d075a07d82399b8cf74b2039bf09caee449405a59"} Oct 31 16:36:58.779429 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:58.779197 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" event={"ID":"82157379-54a9-461f-aa1a-f6e443df2d2d","Type":"ContainerStarted","Data":"59438f0678f90554b9d553e622a5d2353b6ab42fd96819377b1a82c919dd79a7"} Oct 31 16:36:59.933552 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:36:59.933527 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:37:00.084964 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.084877 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-openshift-ca\") pod \"82157379-54a9-461f-aa1a-f6e443df2d2d\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " Oct 31 16:37:00.085118 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.084992 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-xks-ca\") pod \"82157379-54a9-461f-aa1a-f6e443df2d2d\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " Oct 31 16:37:00.085118 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.085086 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82157379-54a9-461f-aa1a-f6e443df2d2d-kube-api-access-lnd77\" (UniqueName: \"kubernetes.io/projected/82157379-54a9-461f-aa1a-f6e443df2d2d-kube-api-access-lnd77\") pod \"82157379-54a9-461f-aa1a-f6e443df2d2d\" (UID: \"82157379-54a9-461f-aa1a-f6e443df2d2d\") " Oct 31 16:37:00.085249 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.085219 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-openshift-ca" pod "82157379-54a9-461f-aa1a-f6e443df2d2d" (UID: "82157379-54a9-461f-aa1a-f6e443df2d2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:00.085423 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.085405 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:00.087240 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.087216 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-xks-ca" pod "82157379-54a9-461f-aa1a-f6e443df2d2d" (UID: "82157379-54a9-461f-aa1a-f6e443df2d2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:00.087381 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.087239 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82157379-54a9-461f-aa1a-f6e443df2d2d-kube-api-access-lnd77" pod "82157379-54a9-461f-aa1a-f6e443df2d2d" (UID: "82157379-54a9-461f-aa1a-f6e443df2d2d"). InnerVolumeSpecName "kube-api-access-lnd77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:00.186275 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.186253 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnd77\" (UniqueName: \"kubernetes.io/projected/82157379-54a9-461f-aa1a-f6e443df2d2d-kube-api-access-lnd77\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:00.186275 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.186276 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82157379-54a9-461f-aa1a-f6e443df2d2d-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:00.807768 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.807736 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" Oct 31 16:37:00.807989 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.807737 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1431612-dxjx9" event={"ID":"82157379-54a9-461f-aa1a-f6e443df2d2d","Type":"ContainerDied","Data":"59438f0678f90554b9d553e622a5d2353b6ab42fd96819377b1a82c919dd79a7"} Oct 31 16:37:00.807989 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:00.807850 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59438f0678f90554b9d553e622a5d2353b6ab42fd96819377b1a82c919dd79a7" Oct 31 16:37:08.260499 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.260463 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk"] Oct 31 16:37:08.261011 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.260908 4399 cpu_manager.go:414] "containerMap: removing container" podUID="82157379-54a9-461f-aa1a-f6e443df2d2d" containerName="revoke-keys" Oct 31 16:37:08.261011 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.260920 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82157379-54a9-461f-aa1a-f6e443df2d2d" containerName="revoke-keys" Oct 31 16:37:08.261130 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.261016 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="82157379-54a9-461f-aa1a-f6e443df2d2d" containerName="revoke-keys" Oct 31 16:37:08.264503 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.264476 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.321148 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.321114 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk"] Oct 31 16:37:08.364711 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.364665 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:37:08.364865 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.364791 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:37:08.371232 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.371207 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2f6cb25-zx4vk\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.371334 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.371240 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb766\" (UniqueName: \"kubernetes.io/projected/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-kube-api-access-fb766\") pod \"maas-api-revoke-sub-d2f6cb25-zx4vk\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.371432 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.371414 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2f6cb25-zx4vk\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.472914 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.472877 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2f6cb25-zx4vk\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.472914 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.472915 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb766\" (UniqueName: \"kubernetes.io/projected/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-kube-api-access-fb766\") pod \"maas-api-revoke-sub-d2f6cb25-zx4vk\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.473115 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.472993 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2f6cb25-zx4vk\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.473636 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.473616 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2f6cb25-zx4vk\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.475488 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.475471 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2f6cb25-zx4vk\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.529030 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.528960 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb766\" (UniqueName: \"kubernetes.io/projected/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-kube-api-access-fb766\") pod \"maas-api-revoke-sub-d2f6cb25-zx4vk\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.577900 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.577871 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:08.949711 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:08.949671 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk"] Oct 31 16:37:08.950982 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:37:08.950954 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb82c93_ba0a_4b08_9336_8b2f8fccd2db.slice/crio-ee11662b24ff504b2b541026725eea64beeee4c26f065ffcbb6fd0be0b36d8be WatchSource:0}: Error finding container ee11662b24ff504b2b541026725eea64beeee4c26f065ffcbb6fd0be0b36d8be: Status 404 returned error can't find the container with id ee11662b24ff504b2b541026725eea64beeee4c26f065ffcbb6fd0be0b36d8be Oct 31 16:37:09.852412 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:09.852378 4399 generic.go:356] "Generic (PLEG): container finished" podID="7bb82c93-ba0a-4b08-9336-8b2f8fccd2db" containerID="75cbabe239a4e176866d382ef26059a445100bf07a0c55d23e8b66c2e8a42b59" exitCode=0 Oct 31 16:37:09.852798 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:09.852460 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" event={"ID":"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db","Type":"ContainerDied","Data":"75cbabe239a4e176866d382ef26059a445100bf07a0c55d23e8b66c2e8a42b59"} Oct 31 16:37:09.852798 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:09.852509 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" event={"ID":"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db","Type":"ContainerStarted","Data":"ee11662b24ff504b2b541026725eea64beeee4c26f065ffcbb6fd0be0b36d8be"} Oct 31 16:37:10.999901 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:10.999878 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:11.097257 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.097226 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-xks-ca\") pod \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " Oct 31 16:37:11.097438 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.097292 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-openshift-ca\") pod \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " Oct 31 16:37:11.097490 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.097432 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-kube-api-access-fb766\" (UniqueName: \"kubernetes.io/projected/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-kube-api-access-fb766\") pod \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\" (UID: \"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db\") " Oct 31 16:37:11.097715 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.097678 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-openshift-ca" pod "7bb82c93-ba0a-4b08-9336-8b2f8fccd2db" (UID: "7bb82c93-ba0a-4b08-9336-8b2f8fccd2db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:11.097836 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.097812 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:11.099439 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.099412 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-xks-ca" pod "7bb82c93-ba0a-4b08-9336-8b2f8fccd2db" (UID: "7bb82c93-ba0a-4b08-9336-8b2f8fccd2db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:11.099747 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.099720 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-kube-api-access-fb766" pod "7bb82c93-ba0a-4b08-9336-8b2f8fccd2db" (UID: "7bb82c93-ba0a-4b08-9336-8b2f8fccd2db"). InnerVolumeSpecName "kube-api-access-fb766". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:11.198954 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.198930 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb766\" (UniqueName: \"kubernetes.io/projected/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-kube-api-access-fb766\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:11.198954 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.198956 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bb82c93-ba0a-4b08-9336-8b2f8fccd2db-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:11.864461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.864427 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" event={"ID":"7bb82c93-ba0a-4b08-9336-8b2f8fccd2db","Type":"ContainerDied","Data":"ee11662b24ff504b2b541026725eea64beeee4c26f065ffcbb6fd0be0b36d8be"} Oct 31 16:37:11.864461 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.864448 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2f6cb25-zx4vk" Oct 31 16:37:11.864683 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:11.864508 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee11662b24ff504b2b541026725eea64beeee4c26f065ffcbb6fd0be0b36d8be" Oct 31 16:37:18.970869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:18.970838 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn"] Oct 31 16:37:18.971274 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:18.971262 4399 cpu_manager.go:414] "containerMap: removing container" podUID="7bb82c93-ba0a-4b08-9336-8b2f8fccd2db" containerName="revoke-keys" Oct 31 16:37:18.971315 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:18.971276 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb82c93-ba0a-4b08-9336-8b2f8fccd2db" containerName="revoke-keys" Oct 31 16:37:18.971367 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:18.971359 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb82c93-ba0a-4b08-9336-8b2f8fccd2db" containerName="revoke-keys" Oct 31 16:37:18.974675 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:18.974654 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.031273 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.031244 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn"] Oct 31 16:37:19.062239 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.062207 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfebeab1-7svjn\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.062403 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.062249 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzd8q\" (UniqueName: \"kubernetes.io/projected/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-kube-api-access-nzd8q\") pod \"maas-api-revoke-sub-cfebeab1-7svjn\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.062403 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.062385 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfebeab1-7svjn\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.076142 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.076124 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:37:19.076253 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.076140 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:37:19.163438 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.163412 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfebeab1-7svjn\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.163582 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.163446 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzd8q\" (UniqueName: \"kubernetes.io/projected/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-kube-api-access-nzd8q\") pod \"maas-api-revoke-sub-cfebeab1-7svjn\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.163582 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.163502 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfebeab1-7svjn\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.164052 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.164026 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfebeab1-7svjn\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.165933 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.165908 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfebeab1-7svjn\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.219357 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.219335 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzd8q\" (UniqueName: \"kubernetes.io/projected/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-kube-api-access-nzd8q\") pod \"maas-api-revoke-sub-cfebeab1-7svjn\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.291736 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.291663 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:19.663470 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.663438 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn"] Oct 31 16:37:19.664649 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:37:19.664626 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7fb18cb_3a49_4d2a_a032_6a433eb3764f.slice/crio-c002a8ac31a979db1036983f82007ed44e2f9f812af1a826b66f75653677fc4e WatchSource:0}: Error finding container c002a8ac31a979db1036983f82007ed44e2f9f812af1a826b66f75653677fc4e: Status 404 returned error can't find the container with id c002a8ac31a979db1036983f82007ed44e2f9f812af1a826b66f75653677fc4e Oct 31 16:37:19.936249 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:19.936176 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" event={"ID":"c7fb18cb-3a49-4d2a-a032-6a433eb3764f","Type":"ContainerStarted","Data":"c002a8ac31a979db1036983f82007ed44e2f9f812af1a826b66f75653677fc4e"} Oct 31 16:37:20.942634 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:20.942550 4399 generic.go:356] "Generic (PLEG): container finished" podID="c7fb18cb-3a49-4d2a-a032-6a433eb3764f" containerID="3d6f0b60c39729d1d2c6c768207b312d6b3e69c9e74923b1008ec2503cbb7a78" exitCode=0 Oct 31 16:37:20.943007 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:20.942624 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" event={"ID":"c7fb18cb-3a49-4d2a-a032-6a433eb3764f","Type":"ContainerDied","Data":"3d6f0b60c39729d1d2c6c768207b312d6b3e69c9e74923b1008ec2503cbb7a78"} Oct 31 16:37:22.091264 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.091235 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:22.189603 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.189581 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-openshift-ca\") pod \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " Oct 31 16:37:22.189761 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.189638 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-xks-ca\") pod \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " Oct 31 16:37:22.189761 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.189676 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-kube-api-access-nzd8q\" (UniqueName: \"kubernetes.io/projected/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-kube-api-access-nzd8q\") pod \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\" (UID: \"c7fb18cb-3a49-4d2a-a032-6a433eb3764f\") " Oct 31 16:37:22.189966 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.189944 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-openshift-ca" pod "c7fb18cb-3a49-4d2a-a032-6a433eb3764f" (UID: "c7fb18cb-3a49-4d2a-a032-6a433eb3764f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:22.191750 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.191730 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-xks-ca" pod "c7fb18cb-3a49-4d2a-a032-6a433eb3764f" (UID: "c7fb18cb-3a49-4d2a-a032-6a433eb3764f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:22.191891 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.191876 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-kube-api-access-nzd8q" pod "c7fb18cb-3a49-4d2a-a032-6a433eb3764f" (UID: "c7fb18cb-3a49-4d2a-a032-6a433eb3764f"). InnerVolumeSpecName "kube-api-access-nzd8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:22.291083 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.291005 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:22.291083 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.291034 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:22.291083 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.291047 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzd8q\" (UniqueName: \"kubernetes.io/projected/c7fb18cb-3a49-4d2a-a032-6a433eb3764f-kube-api-access-nzd8q\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:22.953649 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.953623 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" Oct 31 16:37:22.953649 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.953629 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfebeab1-7svjn" event={"ID":"c7fb18cb-3a49-4d2a-a032-6a433eb3764f","Type":"ContainerDied","Data":"c002a8ac31a979db1036983f82007ed44e2f9f812af1a826b66f75653677fc4e"} Oct 31 16:37:22.953849 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:22.953675 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c002a8ac31a979db1036983f82007ed44e2f9f812af1a826b66f75653677fc4e" Oct 31 16:37:26.206085 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:37:26.206046 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:37:29.939433 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:29.939390 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g"] Oct 31 16:37:29.941898 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:29.939852 4399 cpu_manager.go:414] "containerMap: removing container" podUID="c7fb18cb-3a49-4d2a-a032-6a433eb3764f" containerName="revoke-keys" Oct 31 16:37:29.941898 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:29.939865 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7fb18cb-3a49-4d2a-a032-6a433eb3764f" containerName="revoke-keys" Oct 31 16:37:29.941898 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:29.939929 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7fb18cb-3a49-4d2a-a032-6a433eb3764f" containerName="revoke-keys" Oct 31 16:37:29.943074 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:29.943059 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:29.995524 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:29.995494 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:37:29.996424 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:29.996320 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:37:30.003506 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.003482 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g"] Oct 31 16:37:30.064594 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.064564 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae20d120-f4q4g\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.064818 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.064612 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae20d120-f4q4g\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.064818 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.064738 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm468\" (UniqueName: \"kubernetes.io/projected/52d73349-d45f-4237-b0d5-a59632e1e69d-kube-api-access-sm468\") pod \"maas-api-revoke-sub-ae20d120-f4q4g\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.166106 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.166072 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm468\" (UniqueName: \"kubernetes.io/projected/52d73349-d45f-4237-b0d5-a59632e1e69d-kube-api-access-sm468\") pod \"maas-api-revoke-sub-ae20d120-f4q4g\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.166276 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.166175 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae20d120-f4q4g\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.166276 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.166222 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae20d120-f4q4g\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.166820 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.166798 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae20d120-f4q4g\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.168745 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.168726 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae20d120-f4q4g\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.222239 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.222217 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm468\" (UniqueName: \"kubernetes.io/projected/52d73349-d45f-4237-b0d5-a59632e1e69d-kube-api-access-sm468\") pod \"maas-api-revoke-sub-ae20d120-f4q4g\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.257334 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.257304 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:30.630235 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.630201 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g"] Oct 31 16:37:30.630739 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:37:30.630689 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d73349_d45f_4237_b0d5_a59632e1e69d.slice/crio-1329d0c2a74abb15df984cabe0897f75861e2b7db1e1be0714430cfd40b26dea WatchSource:0}: Error finding container 1329d0c2a74abb15df984cabe0897f75861e2b7db1e1be0714430cfd40b26dea: Status 404 returned error can't find the container with id 1329d0c2a74abb15df984cabe0897f75861e2b7db1e1be0714430cfd40b26dea Oct 31 16:37:30.994927 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.994890 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" event={"ID":"52d73349-d45f-4237-b0d5-a59632e1e69d","Type":"ContainerStarted","Data":"fbf4b1a7c8605bb4855a1ad83e8aa4331dad844b58526ed6396a61880586cb72"} Oct 31 16:37:30.995269 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:30.994956 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" event={"ID":"52d73349-d45f-4237-b0d5-a59632e1e69d","Type":"ContainerStarted","Data":"1329d0c2a74abb15df984cabe0897f75861e2b7db1e1be0714430cfd40b26dea"} Oct 31 16:37:31.114602 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:31.114549 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" podStartSLOduration=2.114535783 podStartE2EDuration="2.114535783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:37:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:37:31.110620938 +0000 UTC m=+6694.321862353" watchObservedRunningTime="2026-10-31 16:37:31.114535783 +0000 UTC m=+6694.325777195" Oct 31 16:37:32.001591 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:32.001558 4399 generic.go:356] "Generic (PLEG): container finished" podID="52d73349-d45f-4237-b0d5-a59632e1e69d" containerID="fbf4b1a7c8605bb4855a1ad83e8aa4331dad844b58526ed6396a61880586cb72" exitCode=0 Oct 31 16:37:32.002012 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:32.001640 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" event={"ID":"52d73349-d45f-4237-b0d5-a59632e1e69d","Type":"ContainerDied","Data":"fbf4b1a7c8605bb4855a1ad83e8aa4331dad844b58526ed6396a61880586cb72"} Oct 31 16:37:33.192216 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.192187 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:33.298554 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.298462 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-openshift-ca\") pod \"52d73349-d45f-4237-b0d5-a59632e1e69d\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " Oct 31 16:37:33.298756 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.298552 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-xks-ca\") pod \"52d73349-d45f-4237-b0d5-a59632e1e69d\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " Oct 31 16:37:33.298756 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.298671 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d73349-d45f-4237-b0d5-a59632e1e69d-kube-api-access-sm468\" (UniqueName: \"kubernetes.io/projected/52d73349-d45f-4237-b0d5-a59632e1e69d-kube-api-access-sm468\") pod \"52d73349-d45f-4237-b0d5-a59632e1e69d\" (UID: \"52d73349-d45f-4237-b0d5-a59632e1e69d\") " Oct 31 16:37:33.298973 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.298942 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-openshift-ca" pod "52d73349-d45f-4237-b0d5-a59632e1e69d" (UID: "52d73349-d45f-4237-b0d5-a59632e1e69d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:33.301012 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.300978 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d73349-d45f-4237-b0d5-a59632e1e69d-kube-api-access-sm468" pod "52d73349-d45f-4237-b0d5-a59632e1e69d" (UID: "52d73349-d45f-4237-b0d5-a59632e1e69d"). InnerVolumeSpecName "kube-api-access-sm468". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:33.301409 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.301383 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-xks-ca" pod "52d73349-d45f-4237-b0d5-a59632e1e69d" (UID: "52d73349-d45f-4237-b0d5-a59632e1e69d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:33.400493 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.400450 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm468\" (UniqueName: \"kubernetes.io/projected/52d73349-d45f-4237-b0d5-a59632e1e69d-kube-api-access-sm468\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.400493 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.400494 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:33.400715 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:33.400512 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d73349-d45f-4237-b0d5-a59632e1e69d-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:34.021155 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:34.021113 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" event={"ID":"52d73349-d45f-4237-b0d5-a59632e1e69d","Type":"ContainerDied","Data":"1329d0c2a74abb15df984cabe0897f75861e2b7db1e1be0714430cfd40b26dea"} Oct 31 16:37:34.021391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:34.021219 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1329d0c2a74abb15df984cabe0897f75861e2b7db1e1be0714430cfd40b26dea" Oct 31 16:37:34.021391 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:34.021311 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae20d120-f4q4g" Oct 31 16:37:34.921917 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:34.921868 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:37:34.922336 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:34.921932 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:37:40.697062 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.696982 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms"] Oct 31 16:37:40.697485 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.697470 4399 cpu_manager.go:414] "containerMap: removing container" podUID="52d73349-d45f-4237-b0d5-a59632e1e69d" containerName="revoke-keys" Oct 31 16:37:40.697530 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.697488 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d73349-d45f-4237-b0d5-a59632e1e69d" containerName="revoke-keys" Oct 31 16:37:40.697581 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.697572 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d73349-d45f-4237-b0d5-a59632e1e69d" containerName="revoke-keys" Oct 31 16:37:40.700775 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.700758 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:40.757710 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.757668 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms"] Oct 31 16:37:40.780774 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.780740 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-187a860f-ph2ms\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:40.780921 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.780856 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-187a860f-ph2ms\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:40.780994 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.780961 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29lk\" (UniqueName: \"kubernetes.io/projected/48671135-10fb-48a2-b6a6-ca276fdaa76d-kube-api-access-m29lk\") pod \"maas-api-revoke-sub-187a860f-ph2ms\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:40.802409 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.802383 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:37:40.802522 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.802446 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:37:40.881869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.881842 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29lk\" (UniqueName: \"kubernetes.io/projected/48671135-10fb-48a2-b6a6-ca276fdaa76d-kube-api-access-m29lk\") pod \"maas-api-revoke-sub-187a860f-ph2ms\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:40.881994 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.881884 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-187a860f-ph2ms\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:40.881994 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.881939 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-187a860f-ph2ms\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:40.882627 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.882606 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-187a860f-ph2ms\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:40.884327 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.884299 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-187a860f-ph2ms\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:40.938510 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:40.938476 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29lk\" (UniqueName: \"kubernetes.io/projected/48671135-10fb-48a2-b6a6-ca276fdaa76d-kube-api-access-m29lk\") pod \"maas-api-revoke-sub-187a860f-ph2ms\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:41.013748 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:41.013677 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:41.389472 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:41.389443 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms"] Oct 31 16:37:41.389768 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:37:41.389743 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48671135_10fb_48a2_b6a6_ca276fdaa76d.slice/crio-b29cbe2c4c0f653ca556a7873092c3baef43f7b078bf58fdcd68cf0b4cb4f98d WatchSource:0}: Error finding container b29cbe2c4c0f653ca556a7873092c3baef43f7b078bf58fdcd68cf0b4cb4f98d: Status 404 returned error can't find the container with id b29cbe2c4c0f653ca556a7873092c3baef43f7b078bf58fdcd68cf0b4cb4f98d Oct 31 16:37:42.067771 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:42.067735 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" event={"ID":"48671135-10fb-48a2-b6a6-ca276fdaa76d","Type":"ContainerStarted","Data":"8775b207b357eee4a695e8e2ca029de22fb4302254bb7796dd112da5b8f5ea04"} Oct 31 16:37:42.068238 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:42.067791 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" event={"ID":"48671135-10fb-48a2-b6a6-ca276fdaa76d","Type":"ContainerStarted","Data":"b29cbe2c4c0f653ca556a7873092c3baef43f7b078bf58fdcd68cf0b4cb4f98d"} Oct 31 16:37:42.180322 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:42.180268 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" podStartSLOduration=2.180252465 podStartE2EDuration="2.180252465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:37:42.179548958 +0000 UTC m=+6705.390790393" watchObservedRunningTime="2026-10-31 16:37:42.180252465 +0000 UTC m=+6705.391493885" Oct 31 16:37:43.074847 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:43.074809 4399 generic.go:356] "Generic (PLEG): container finished" podID="48671135-10fb-48a2-b6a6-ca276fdaa76d" containerID="8775b207b357eee4a695e8e2ca029de22fb4302254bb7796dd112da5b8f5ea04" exitCode=0 Oct 31 16:37:43.075289 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:43.074856 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" event={"ID":"48671135-10fb-48a2-b6a6-ca276fdaa76d","Type":"ContainerDied","Data":"8775b207b357eee4a695e8e2ca029de22fb4302254bb7796dd112da5b8f5ea04"} Oct 31 16:37:44.267877 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.267848 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:44.320416 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.320369 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48671135-10fb-48a2-b6a6-ca276fdaa76d-kube-api-access-m29lk\" (UniqueName: \"kubernetes.io/projected/48671135-10fb-48a2-b6a6-ca276fdaa76d-kube-api-access-m29lk\") pod \"48671135-10fb-48a2-b6a6-ca276fdaa76d\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " Oct 31 16:37:44.320618 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.320499 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-openshift-ca\") pod \"48671135-10fb-48a2-b6a6-ca276fdaa76d\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " Oct 31 16:37:44.320618 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.320582 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-xks-ca\") pod \"48671135-10fb-48a2-b6a6-ca276fdaa76d\" (UID: \"48671135-10fb-48a2-b6a6-ca276fdaa76d\") " Oct 31 16:37:44.320997 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.320966 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-openshift-ca" pod "48671135-10fb-48a2-b6a6-ca276fdaa76d" (UID: "48671135-10fb-48a2-b6a6-ca276fdaa76d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:44.322809 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.322774 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48671135-10fb-48a2-b6a6-ca276fdaa76d-kube-api-access-m29lk" pod "48671135-10fb-48a2-b6a6-ca276fdaa76d" (UID: "48671135-10fb-48a2-b6a6-ca276fdaa76d"). InnerVolumeSpecName "kube-api-access-m29lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:44.323333 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.323307 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-xks-ca" pod "48671135-10fb-48a2-b6a6-ca276fdaa76d" (UID: "48671135-10fb-48a2-b6a6-ca276fdaa76d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:44.469722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.454325 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m29lk\" (UniqueName: \"kubernetes.io/projected/48671135-10fb-48a2-b6a6-ca276fdaa76d-kube-api-access-m29lk\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:44.469722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.454363 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:44.469722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:44.454380 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48671135-10fb-48a2-b6a6-ca276fdaa76d-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:45.119725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:45.116088 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" event={"ID":"48671135-10fb-48a2-b6a6-ca276fdaa76d","Type":"ContainerDied","Data":"b29cbe2c4c0f653ca556a7873092c3baef43f7b078bf58fdcd68cf0b4cb4f98d"} Oct 31 16:37:45.119725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:45.116150 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b29cbe2c4c0f653ca556a7873092c3baef43f7b078bf58fdcd68cf0b4cb4f98d" Oct 31 16:37:45.119725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:45.116224 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-187a860f-ph2ms" Oct 31 16:37:51.767326 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.767288 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj"] Oct 31 16:37:51.768038 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.768007 4399 cpu_manager.go:414] "containerMap: removing container" podUID="48671135-10fb-48a2-b6a6-ca276fdaa76d" containerName="revoke-keys" Oct 31 16:37:51.768119 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.768042 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48671135-10fb-48a2-b6a6-ca276fdaa76d" containerName="revoke-keys" Oct 31 16:37:51.768158 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.768142 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="48671135-10fb-48a2-b6a6-ca276fdaa76d" containerName="revoke-keys" Oct 31 16:37:51.771429 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.771411 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:51.826277 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.826250 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj"] Oct 31 16:37:51.868814 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.868782 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7426273c-hlzcj\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:51.868941 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.868820 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7ggz\" (UniqueName: \"kubernetes.io/projected/b8237144-e60c-4b26-9122-daa6afa9939b-kube-api-access-c7ggz\") pod \"maas-api-revoke-sub-7426273c-hlzcj\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:51.868941 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.868919 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7426273c-hlzcj\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:51.870471 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.870447 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:37:51.870579 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.870500 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:37:51.969909 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.969881 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7426273c-hlzcj\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:51.970669 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.970296 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7426273c-hlzcj\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:51.970669 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.970378 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7ggz\" (UniqueName: \"kubernetes.io/projected/b8237144-e60c-4b26-9122-daa6afa9939b-kube-api-access-c7ggz\") pod \"maas-api-revoke-sub-7426273c-hlzcj\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:51.971014 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.970992 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7426273c-hlzcj\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:51.972561 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:51.972539 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7426273c-hlzcj\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:52.026357 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:52.026292 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7ggz\" (UniqueName: \"kubernetes.io/projected/b8237144-e60c-4b26-9122-daa6afa9939b-kube-api-access-c7ggz\") pod \"maas-api-revoke-sub-7426273c-hlzcj\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:52.084186 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:52.084164 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:52.463739 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:52.459987 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj"] Oct 31 16:37:52.465871 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:37:52.465834 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8237144_e60c_4b26_9122_daa6afa9939b.slice/crio-95b9fb03a0e58eb5811bc03f754377585e0808a2928933354290ca6ab1c30141 WatchSource:0}: Error finding container 95b9fb03a0e58eb5811bc03f754377585e0808a2928933354290ca6ab1c30141: Status 404 returned error can't find the container with id 95b9fb03a0e58eb5811bc03f754377585e0808a2928933354290ca6ab1c30141 Oct 31 16:37:53.170987 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:53.170900 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" event={"ID":"b8237144-e60c-4b26-9122-daa6afa9939b","Type":"ContainerStarted","Data":"585662211a93e3884e0e6d8a6feec1f9f40d96457362e1cf2cdcde277e868a75"} Oct 31 16:37:53.170987 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:53.170952 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" event={"ID":"b8237144-e60c-4b26-9122-daa6afa9939b","Type":"ContainerStarted","Data":"95b9fb03a0e58eb5811bc03f754377585e0808a2928933354290ca6ab1c30141"} Oct 31 16:37:53.284839 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:53.284789 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" podStartSLOduration=2.284775097 podStartE2EDuration="2.284775097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:37:53.282261214 +0000 UTC m=+6716.493502627" watchObservedRunningTime="2026-10-31 16:37:53.284775097 +0000 UTC m=+6716.496016511" Oct 31 16:37:54.179460 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:54.179419 4399 generic.go:356] "Generic (PLEG): container finished" podID="b8237144-e60c-4b26-9122-daa6afa9939b" containerID="585662211a93e3884e0e6d8a6feec1f9f40d96457362e1cf2cdcde277e868a75" exitCode=0 Oct 31 16:37:54.179893 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:54.179504 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" event={"ID":"b8237144-e60c-4b26-9122-daa6afa9939b","Type":"ContainerDied","Data":"585662211a93e3884e0e6d8a6feec1f9f40d96457362e1cf2cdcde277e868a75"} Oct 31 16:37:55.619380 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.619042 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:55.728159 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.728127 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-xks-ca\") pod \"b8237144-e60c-4b26-9122-daa6afa9939b\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " Oct 31 16:37:55.728352 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.728177 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-openshift-ca\") pod \"b8237144-e60c-4b26-9122-daa6afa9939b\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " Oct 31 16:37:55.728352 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.728323 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8237144-e60c-4b26-9122-daa6afa9939b-kube-api-access-c7ggz\" (UniqueName: \"kubernetes.io/projected/b8237144-e60c-4b26-9122-daa6afa9939b-kube-api-access-c7ggz\") pod \"b8237144-e60c-4b26-9122-daa6afa9939b\" (UID: \"b8237144-e60c-4b26-9122-daa6afa9939b\") " Oct 31 16:37:55.728607 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.728576 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-openshift-ca" pod "b8237144-e60c-4b26-9122-daa6afa9939b" (UID: "b8237144-e60c-4b26-9122-daa6afa9939b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:37:55.728769 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.728751 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:55.730546 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.730507 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-xks-ca" pod "b8237144-e60c-4b26-9122-daa6afa9939b" (UID: "b8237144-e60c-4b26-9122-daa6afa9939b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:37:55.730642 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.730514 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8237144-e60c-4b26-9122-daa6afa9939b-kube-api-access-c7ggz" pod "b8237144-e60c-4b26-9122-daa6afa9939b" (UID: "b8237144-e60c-4b26-9122-daa6afa9939b"). InnerVolumeSpecName "kube-api-access-c7ggz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:37:55.829666 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.829574 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8237144-e60c-4b26-9122-daa6afa9939b-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:55.829666 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:55.829607 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7ggz\" (UniqueName: \"kubernetes.io/projected/b8237144-e60c-4b26-9122-daa6afa9939b-kube-api-access-c7ggz\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:37:56.217724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:56.216856 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" Oct 31 16:37:56.217724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:56.216935 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7426273c-hlzcj" event={"ID":"b8237144-e60c-4b26-9122-daa6afa9939b","Type":"ContainerDied","Data":"95b9fb03a0e58eb5811bc03f754377585e0808a2928933354290ca6ab1c30141"} Oct 31 16:37:56.217724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:56.217013 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b9fb03a0e58eb5811bc03f754377585e0808a2928933354290ca6ab1c30141" Oct 31 16:37:56.253803 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:37:56.253664 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:37:56.272728 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:56.268816 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-vfnzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:37:56.272728 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:37:56.268872 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-vfnzr" podUID="c09561dd-b9cb-49e2-841d-54d528822afb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:38:02.727460 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.727423 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v"] Oct 31 16:38:02.728426 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.728066 4399 cpu_manager.go:414] "containerMap: removing container" podUID="b8237144-e60c-4b26-9122-daa6afa9939b" containerName="revoke-keys" Oct 31 16:38:02.728426 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.728086 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8237144-e60c-4b26-9122-daa6afa9939b" containerName="revoke-keys" Oct 31 16:38:02.728426 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.728253 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8237144-e60c-4b26-9122-daa6afa9939b" containerName="revoke-keys" Oct 31 16:38:02.731957 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.731936 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:02.786432 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.786404 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v"] Oct 31 16:38:02.815495 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.815472 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650796a4-lz98v\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:02.815637 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.815525 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6j57\" (UniqueName: \"kubernetes.io/projected/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-kube-api-access-p6j57\") pod \"maas-api-revoke-sub-650796a4-lz98v\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:02.815637 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.815562 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650796a4-lz98v\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:02.831666 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.831637 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:38:02.831808 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.831689 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:38:02.917066 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.917040 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6j57\" (UniqueName: \"kubernetes.io/projected/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-kube-api-access-p6j57\") pod \"maas-api-revoke-sub-650796a4-lz98v\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:02.917224 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.917077 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650796a4-lz98v\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:02.917224 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.917133 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650796a4-lz98v\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:02.917724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.917684 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650796a4-lz98v\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:02.919584 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.919564 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650796a4-lz98v\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:02.973586 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:02.973553 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6j57\" (UniqueName: \"kubernetes.io/projected/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-kube-api-access-p6j57\") pod \"maas-api-revoke-sub-650796a4-lz98v\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:03.050930 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:03.050876 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:03.428394 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:03.428359 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v"] Oct 31 16:38:03.428641 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:38:03.428620 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1401eb6_f43f_45f5_a1be_dddfb77a4c3a.slice/crio-a20bb7805941597a94523b9a8a1a62198fea3c3d0bf0feafca0f905e5164aac4 WatchSource:0}: Error finding container a20bb7805941597a94523b9a8a1a62198fea3c3d0bf0feafca0f905e5164aac4: Status 404 returned error can't find the container with id a20bb7805941597a94523b9a8a1a62198fea3c3d0bf0feafca0f905e5164aac4 Oct 31 16:38:04.269075 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:04.269036 4399 generic.go:356] "Generic (PLEG): container finished" podID="f1401eb6-f43f-45f5-a1be-dddfb77a4c3a" containerID="a02c7a39fb26ffbde09f8e55d73afd2b9bf980213b41bda3fa137ae1e548d573" exitCode=0 Oct 31 16:38:04.269423 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:04.269117 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" event={"ID":"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a","Type":"ContainerDied","Data":"a02c7a39fb26ffbde09f8e55d73afd2b9bf980213b41bda3fa137ae1e548d573"} Oct 31 16:38:04.269423 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:04.269215 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" event={"ID":"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a","Type":"ContainerStarted","Data":"a20bb7805941597a94523b9a8a1a62198fea3c3d0bf0feafca0f905e5164aac4"} Oct 31 16:38:05.449422 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.449382 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:05.541772 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.541644 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-xks-ca\") pod \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " Oct 31 16:38:05.541772 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.541744 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-kube-api-access-p6j57\" (UniqueName: \"kubernetes.io/projected/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-kube-api-access-p6j57\") pod \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " Oct 31 16:38:05.542037 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.541847 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-openshift-ca\") pod \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\" (UID: \"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a\") " Oct 31 16:38:05.542225 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.542193 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-openshift-ca" pod "f1401eb6-f43f-45f5-a1be-dddfb77a4c3a" (UID: "f1401eb6-f43f-45f5-a1be-dddfb77a4c3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:38:05.542371 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.542291 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:05.544772 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.544735 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-kube-api-access-p6j57" pod "f1401eb6-f43f-45f5-a1be-dddfb77a4c3a" (UID: "f1401eb6-f43f-45f5-a1be-dddfb77a4c3a"). InnerVolumeSpecName "kube-api-access-p6j57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:38:05.544929 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.544759 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-xks-ca" pod "f1401eb6-f43f-45f5-a1be-dddfb77a4c3a" (UID: "f1401eb6-f43f-45f5-a1be-dddfb77a4c3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:38:05.643846 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.643807 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:05.643846 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:05.643848 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6j57\" (UniqueName: \"kubernetes.io/projected/f1401eb6-f43f-45f5-a1be-dddfb77a4c3a-kube-api-access-p6j57\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:06.312047 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:06.312007 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" event={"ID":"f1401eb6-f43f-45f5-a1be-dddfb77a4c3a","Type":"ContainerDied","Data":"a20bb7805941597a94523b9a8a1a62198fea3c3d0bf0feafca0f905e5164aac4"} Oct 31 16:38:06.312358 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:06.312335 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a20bb7805941597a94523b9a8a1a62198fea3c3d0bf0feafca0f905e5164aac4" Oct 31 16:38:06.312521 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:06.312214 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650796a4-lz98v" Oct 31 16:38:13.485665 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.485623 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q"] Oct 31 16:38:13.486084 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.486068 4399 cpu_manager.go:414] "containerMap: removing container" podUID="f1401eb6-f43f-45f5-a1be-dddfb77a4c3a" containerName="revoke-keys" Oct 31 16:38:13.486084 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.486084 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1401eb6-f43f-45f5-a1be-dddfb77a4c3a" containerName="revoke-keys" Oct 31 16:38:13.486189 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.486180 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1401eb6-f43f-45f5-a1be-dddfb77a4c3a" containerName="revoke-keys" Oct 31 16:38:13.489588 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.489572 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.545182 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.545153 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q"] Oct 31 16:38:13.565491 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.565456 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj5ww\" (UniqueName: \"kubernetes.io/projected/58a78b86-975c-4330-a195-e73bccef03f2-kube-api-access-kj5ww\") pod \"maas-api-revoke-sub-655f22d7-c2d9q\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.565652 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.565506 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655f22d7-c2d9q\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.565652 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.565637 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655f22d7-c2d9q\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.590085 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.590062 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:38:13.590235 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.590081 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:38:13.666474 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.666445 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj5ww\" (UniqueName: \"kubernetes.io/projected/58a78b86-975c-4330-a195-e73bccef03f2-kube-api-access-kj5ww\") pod \"maas-api-revoke-sub-655f22d7-c2d9q\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.666612 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.666501 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655f22d7-c2d9q\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.666612 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.666579 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655f22d7-c2d9q\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.667140 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.667116 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-655f22d7-c2d9q\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.669058 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.669036 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-655f22d7-c2d9q\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.722688 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.722665 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj5ww\" (UniqueName: \"kubernetes.io/projected/58a78b86-975c-4330-a195-e73bccef03f2-kube-api-access-kj5ww\") pod \"maas-api-revoke-sub-655f22d7-c2d9q\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:13.807376 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:13.807327 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:14.182940 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:14.182905 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q"] Oct 31 16:38:14.183763 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:38:14.183734 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a78b86_975c_4330_a195_e73bccef03f2.slice/crio-daac3bdf0ac1adf8adfb2b6336d5bea8ba9a6ebdd4b0789ca507e7be574ffff6 WatchSource:0}: Error finding container daac3bdf0ac1adf8adfb2b6336d5bea8ba9a6ebdd4b0789ca507e7be574ffff6: Status 404 returned error can't find the container with id daac3bdf0ac1adf8adfb2b6336d5bea8ba9a6ebdd4b0789ca507e7be574ffff6 Oct 31 16:38:14.359324 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:14.359287 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" event={"ID":"58a78b86-975c-4330-a195-e73bccef03f2","Type":"ContainerStarted","Data":"daac3bdf0ac1adf8adfb2b6336d5bea8ba9a6ebdd4b0789ca507e7be574ffff6"} Oct 31 16:38:15.366732 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:15.366629 4399 generic.go:356] "Generic (PLEG): container finished" podID="58a78b86-975c-4330-a195-e73bccef03f2" containerID="0cd7d0cc07651efcd146b843e732bd899a9757117b53b14e04518d475e01ba21" exitCode=0 Oct 31 16:38:15.367115 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:15.366732 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" event={"ID":"58a78b86-975c-4330-a195-e73bccef03f2","Type":"ContainerDied","Data":"0cd7d0cc07651efcd146b843e732bd899a9757117b53b14e04518d475e01ba21"} Oct 31 16:38:16.523016 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.522986 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:16.597244 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.597191 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-xks-ca\") pod \"58a78b86-975c-4330-a195-e73bccef03f2\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " Oct 31 16:38:16.597438 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.597316 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-openshift-ca\") pod \"58a78b86-975c-4330-a195-e73bccef03f2\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " Oct 31 16:38:16.597438 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.597392 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a78b86-975c-4330-a195-e73bccef03f2-kube-api-access-kj5ww\" (UniqueName: \"kubernetes.io/projected/58a78b86-975c-4330-a195-e73bccef03f2-kube-api-access-kj5ww\") pod \"58a78b86-975c-4330-a195-e73bccef03f2\" (UID: \"58a78b86-975c-4330-a195-e73bccef03f2\") " Oct 31 16:38:16.598021 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.597987 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-openshift-ca" pod "58a78b86-975c-4330-a195-e73bccef03f2" (UID: "58a78b86-975c-4330-a195-e73bccef03f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:38:16.599849 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.599825 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a78b86-975c-4330-a195-e73bccef03f2-kube-api-access-kj5ww" pod "58a78b86-975c-4330-a195-e73bccef03f2" (UID: "58a78b86-975c-4330-a195-e73bccef03f2"). InnerVolumeSpecName "kube-api-access-kj5ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:38:16.600321 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.600292 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-xks-ca" pod "58a78b86-975c-4330-a195-e73bccef03f2" (UID: "58a78b86-975c-4330-a195-e73bccef03f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:38:16.698488 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.698444 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj5ww\" (UniqueName: \"kubernetes.io/projected/58a78b86-975c-4330-a195-e73bccef03f2-kube-api-access-kj5ww\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:16.698488 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.698475 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:16.698488 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:16.698484 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a78b86-975c-4330-a195-e73bccef03f2-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:17.406724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:17.405855 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" event={"ID":"58a78b86-975c-4330-a195-e73bccef03f2","Type":"ContainerDied","Data":"daac3bdf0ac1adf8adfb2b6336d5bea8ba9a6ebdd4b0789ca507e7be574ffff6"} Oct 31 16:38:17.406724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:17.405970 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daac3bdf0ac1adf8adfb2b6336d5bea8ba9a6ebdd4b0789ca507e7be574ffff6" Oct 31 16:38:17.406724 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:17.406059 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-655f22d7-c2d9q" Oct 31 16:38:24.239890 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.239861 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b"] Oct 31 16:38:24.240313 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.240298 4399 cpu_manager.go:414] "containerMap: removing container" podUID="58a78b86-975c-4330-a195-e73bccef03f2" containerName="revoke-keys" Oct 31 16:38:24.240313 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.240309 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a78b86-975c-4330-a195-e73bccef03f2" containerName="revoke-keys" Oct 31 16:38:24.240403 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.240373 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a78b86-975c-4330-a195-e73bccef03f2" containerName="revoke-keys" Oct 31 16:38:24.243710 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.243676 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.274545 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.274522 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6xn4\" (UniqueName: \"kubernetes.io/projected/9f46de05-d0b4-415b-bc5d-fafc2ea30319-kube-api-access-j6xn4\") pod \"maas-api-revoke-sub-aaa8fc1d-d775b\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.274663 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.274559 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa8fc1d-d775b\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.274731 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.274668 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa8fc1d-d775b\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.299463 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.299437 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b"] Oct 31 16:38:24.343995 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.343955 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:38:24.344164 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.344053 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:38:24.375182 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.375137 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa8fc1d-d775b\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.375340 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.375236 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa8fc1d-d775b\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.375340 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.375287 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6xn4\" (UniqueName: \"kubernetes.io/projected/9f46de05-d0b4-415b-bc5d-fafc2ea30319-kube-api-access-j6xn4\") pod \"maas-api-revoke-sub-aaa8fc1d-d775b\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.375791 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.375772 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaa8fc1d-d775b\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.377666 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.377647 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaa8fc1d-d775b\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.431443 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.431421 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6xn4\" (UniqueName: \"kubernetes.io/projected/9f46de05-d0b4-415b-bc5d-fafc2ea30319-kube-api-access-j6xn4\") pod \"maas-api-revoke-sub-aaa8fc1d-d775b\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.559667 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.559603 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:24.938741 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:24.938712 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b"] Oct 31 16:38:24.939297 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:38:24.939269 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f46de05_d0b4_415b_bc5d_fafc2ea30319.slice/crio-8b515dbcd2832795f35a636ebf56ca19a062f1e72e420130508a31d3d945137d WatchSource:0}: Error finding container 8b515dbcd2832795f35a636ebf56ca19a062f1e72e420130508a31d3d945137d: Status 404 returned error can't find the container with id 8b515dbcd2832795f35a636ebf56ca19a062f1e72e420130508a31d3d945137d Oct 31 16:38:25.045901 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:25.045872 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2"] Oct 31 16:38:25.049241 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:25.049223 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e75faa62-rphj2"] Oct 31 16:38:25.456065 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:25.456034 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" event={"ID":"9f46de05-d0b4-415b-bc5d-fafc2ea30319","Type":"ContainerStarted","Data":"322054811de7237fd08ccdca995e8d04e48aaaae500b5c4b87ceb4994a7cbd22"} Oct 31 16:38:25.456470 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:25.456092 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" event={"ID":"9f46de05-d0b4-415b-bc5d-fafc2ea30319","Type":"ContainerStarted","Data":"8b515dbcd2832795f35a636ebf56ca19a062f1e72e420130508a31d3d945137d"} Oct 31 16:38:25.620160 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:25.620066 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" podStartSLOduration=1.62005081 podStartE2EDuration="1.62005081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:25.568185729 +0000 UTC m=+6748.779427153" watchObservedRunningTime="2026-10-31 16:38:25.62005081 +0000 UTC m=+6748.831292243" Oct 31 16:38:26.208042 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:38:26.208005 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:38:26.462556 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:26.462468 4399 generic.go:356] "Generic (PLEG): container finished" podID="9f46de05-d0b4-415b-bc5d-fafc2ea30319" containerID="322054811de7237fd08ccdca995e8d04e48aaaae500b5c4b87ceb4994a7cbd22" exitCode=0 Oct 31 16:38:26.462973 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:26.462561 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" event={"ID":"9f46de05-d0b4-415b-bc5d-fafc2ea30319","Type":"ContainerDied","Data":"322054811de7237fd08ccdca995e8d04e48aaaae500b5c4b87ceb4994a7cbd22"} Oct 31 16:38:26.896615 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:26.896520 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a" path="/var/lib/kubelet/pods/42dd0f3f-be5d-4c6e-bd80-1f1ede469b3a/volumes" Oct 31 16:38:27.642591 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.642567 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:27.709299 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.709267 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-openshift-ca\") pod \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " Oct 31 16:38:27.709483 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.709331 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-xks-ca\") pod \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " Oct 31 16:38:27.709483 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.709430 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f46de05-d0b4-415b-bc5d-fafc2ea30319-kube-api-access-j6xn4\" (UniqueName: \"kubernetes.io/projected/9f46de05-d0b4-415b-bc5d-fafc2ea30319-kube-api-access-j6xn4\") pod \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\" (UID: \"9f46de05-d0b4-415b-bc5d-fafc2ea30319\") " Oct 31 16:38:27.709685 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.709665 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-openshift-ca" pod "9f46de05-d0b4-415b-bc5d-fafc2ea30319" (UID: "9f46de05-d0b4-415b-bc5d-fafc2ea30319"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:38:27.709886 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.709867 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:27.712102 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.712081 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-xks-ca" pod "9f46de05-d0b4-415b-bc5d-fafc2ea30319" (UID: "9f46de05-d0b4-415b-bc5d-fafc2ea30319"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:38:27.712200 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.712187 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f46de05-d0b4-415b-bc5d-fafc2ea30319-kube-api-access-j6xn4" pod "9f46de05-d0b4-415b-bc5d-fafc2ea30319" (UID: "9f46de05-d0b4-415b-bc5d-fafc2ea30319"). InnerVolumeSpecName "kube-api-access-j6xn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:38:27.811341 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.811303 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f46de05-d0b4-415b-bc5d-fafc2ea30319-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:27.811497 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:27.811344 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6xn4\" (UniqueName: \"kubernetes.io/projected/9f46de05-d0b4-415b-bc5d-fafc2ea30319-kube-api-access-j6xn4\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:28.501498 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:28.501456 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" event={"ID":"9f46de05-d0b4-415b-bc5d-fafc2ea30319","Type":"ContainerDied","Data":"8b515dbcd2832795f35a636ebf56ca19a062f1e72e420130508a31d3d945137d"} Oct 31 16:38:28.501808 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:28.501787 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b515dbcd2832795f35a636ebf56ca19a062f1e72e420130508a31d3d945137d" Oct 31 16:38:28.502020 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:28.502003 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaa8fc1d-d775b" Oct 31 16:38:35.009048 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.009017 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt"] Oct 31 16:38:35.009568 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.009445 4399 cpu_manager.go:414] "containerMap: removing container" podUID="9f46de05-d0b4-415b-bc5d-fafc2ea30319" containerName="revoke-keys" Oct 31 16:38:35.009568 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.009460 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f46de05-d0b4-415b-bc5d-fafc2ea30319" containerName="revoke-keys" Oct 31 16:38:35.009568 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.009549 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f46de05-d0b4-415b-bc5d-fafc2ea30319" containerName="revoke-keys" Oct 31 16:38:35.012578 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.012556 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.069410 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.069378 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt"] Oct 31 16:38:35.081787 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.081759 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vj6m\" (UniqueName: \"kubernetes.io/projected/724e729b-d584-4654-b406-2a2f0f23948d-kube-api-access-6vj6m\") pod \"maas-api-revoke-sub-9d86b2c1-d2skt\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.081942 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.081836 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d86b2c1-d2skt\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.081942 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.081928 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d86b2c1-d2skt\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.114239 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.114209 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:38:35.114371 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.114245 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:38:35.183147 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.183120 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d86b2c1-d2skt\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.183269 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.183155 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d86b2c1-d2skt\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.183269 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.183244 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vj6m\" (UniqueName: \"kubernetes.io/projected/724e729b-d584-4654-b406-2a2f0f23948d-kube-api-access-6vj6m\") pod \"maas-api-revoke-sub-9d86b2c1-d2skt\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.191821 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.184288 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d86b2c1-d2skt\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.191821 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.186760 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d86b2c1-d2skt\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.239726 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.239668 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vj6m\" (UniqueName: \"kubernetes.io/projected/724e729b-d584-4654-b406-2a2f0f23948d-kube-api-access-6vj6m\") pod \"maas-api-revoke-sub-9d86b2c1-d2skt\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.326619 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.326556 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:35.700108 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:35.700081 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt"] Oct 31 16:38:35.701046 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:38:35.701021 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724e729b_d584_4654_b406_2a2f0f23948d.slice/crio-72213f71e07bc8c209f00750e296cf8575a2d6aa42c07c79ce27898f2797002b WatchSource:0}: Error finding container 72213f71e07bc8c209f00750e296cf8575a2d6aa42c07c79ce27898f2797002b: Status 404 returned error can't find the container with id 72213f71e07bc8c209f00750e296cf8575a2d6aa42c07c79ce27898f2797002b Oct 31 16:38:36.045113 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:36.045074 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk"] Oct 31 16:38:36.094079 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:36.094057 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cc4542d-lhnjk"] Oct 31 16:38:36.552453 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:36.552419 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" event={"ID":"724e729b-d584-4654-b406-2a2f0f23948d","Type":"ContainerStarted","Data":"e65fc7664aa16b801286ce6c00da9ddb83cc4ce0df7e47f72eeb28fa0214c546"} Oct 31 16:38:36.552645 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:36.552472 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" event={"ID":"724e729b-d584-4654-b406-2a2f0f23948d","Type":"ContainerStarted","Data":"72213f71e07bc8c209f00750e296cf8575a2d6aa42c07c79ce27898f2797002b"} Oct 31 16:38:36.715947 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:36.715883 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" podStartSLOduration=2.715867379 podStartE2EDuration="2.715867379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 16:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:38:36.66356849 +0000 UTC m=+6759.874809892" watchObservedRunningTime="2026-10-31 16:38:36.715867379 +0000 UTC m=+6759.927108808" Oct 31 16:38:36.897477 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:36.897400 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5695b68a-b305-49d6-8bfd-5bd50e13a2d9" path="/var/lib/kubelet/pods/5695b68a-b305-49d6-8bfd-5bd50e13a2d9/volumes" Oct 31 16:38:37.562831 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:37.562797 4399 generic.go:356] "Generic (PLEG): container finished" podID="724e729b-d584-4654-b406-2a2f0f23948d" containerID="e65fc7664aa16b801286ce6c00da9ddb83cc4ce0df7e47f72eeb28fa0214c546" exitCode=0 Oct 31 16:38:37.563419 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:37.562894 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" event={"ID":"724e729b-d584-4654-b406-2a2f0f23948d","Type":"ContainerDied","Data":"e65fc7664aa16b801286ce6c00da9ddb83cc4ce0df7e47f72eeb28fa0214c546"} Oct 31 16:38:39.026454 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.026418 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:39.133323 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.133295 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724e729b-d584-4654-b406-2a2f0f23948d-kube-api-access-6vj6m\" (UniqueName: \"kubernetes.io/projected/724e729b-d584-4654-b406-2a2f0f23948d-kube-api-access-6vj6m\") pod \"724e729b-d584-4654-b406-2a2f0f23948d\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " Oct 31 16:38:39.133497 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.133360 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-openshift-ca\") pod \"724e729b-d584-4654-b406-2a2f0f23948d\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " Oct 31 16:38:39.133497 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.133459 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-xks-ca\") pod \"724e729b-d584-4654-b406-2a2f0f23948d\" (UID: \"724e729b-d584-4654-b406-2a2f0f23948d\") " Oct 31 16:38:39.134142 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.134095 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-openshift-ca" pod "724e729b-d584-4654-b406-2a2f0f23948d" (UID: "724e729b-d584-4654-b406-2a2f0f23948d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:38:39.135599 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.135577 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724e729b-d584-4654-b406-2a2f0f23948d-kube-api-access-6vj6m" pod "724e729b-d584-4654-b406-2a2f0f23948d" (UID: "724e729b-d584-4654-b406-2a2f0f23948d"). InnerVolumeSpecName "kube-api-access-6vj6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:38:39.135712 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.135676 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-xks-ca" pod "724e729b-d584-4654-b406-2a2f0f23948d" (UID: "724e729b-d584-4654-b406-2a2f0f23948d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:38:39.234597 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.234564 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:39.234597 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.234594 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vj6m\" (UniqueName: \"kubernetes.io/projected/724e729b-d584-4654-b406-2a2f0f23948d-kube-api-access-6vj6m\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:39.234597 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.234604 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724e729b-d584-4654-b406-2a2f0f23948d-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:39.607929 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.607682 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" event={"ID":"724e729b-d584-4654-b406-2a2f0f23948d","Type":"ContainerDied","Data":"72213f71e07bc8c209f00750e296cf8575a2d6aa42c07c79ce27898f2797002b"} Oct 31 16:38:39.607929 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.607765 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72213f71e07bc8c209f00750e296cf8575a2d6aa42c07c79ce27898f2797002b" Oct 31 16:38:39.607929 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:39.607792 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d86b2c1-d2skt" Oct 31 16:38:45.755344 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.755310 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t"] Oct 31 16:38:45.755763 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.755756 4399 cpu_manager.go:414] "containerMap: removing container" podUID="724e729b-d584-4654-b406-2a2f0f23948d" containerName="revoke-keys" Oct 31 16:38:45.755816 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.755768 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724e729b-d584-4654-b406-2a2f0f23948d" containerName="revoke-keys" Oct 31 16:38:45.755869 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.755858 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="724e729b-d584-4654-b406-2a2f0f23948d" containerName="revoke-keys" Oct 31 16:38:45.759272 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.759254 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:45.814653 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.814618 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t"] Oct 31 16:38:45.841268 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.841233 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c66b3f-b658t\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:45.841433 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.841279 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9zpb\" (UniqueName: \"kubernetes.io/projected/87abeeb0-1509-49af-8c1d-1fd128a15f88-kube-api-access-h9zpb\") pod \"maas-api-revoke-sub-89c66b3f-b658t\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:45.841433 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.841403 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c66b3f-b658t\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:45.859575 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.859550 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 16:38:45.859575 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.859564 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qx7jf\"" Oct 31 16:38:45.942850 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.942814 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c66b3f-b658t\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:45.943063 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.942882 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c66b3f-b658t\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:45.943063 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.942914 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9zpb\" (UniqueName: \"kubernetes.io/projected/87abeeb0-1509-49af-8c1d-1fd128a15f88-kube-api-access-h9zpb\") pod \"maas-api-revoke-sub-89c66b3f-b658t\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:45.943447 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.943426 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c66b3f-b658t\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:45.945462 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.945440 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c66b3f-b658t\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:45.999075 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:45.999041 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9zpb\" (UniqueName: \"kubernetes.io/projected/87abeeb0-1509-49af-8c1d-1fd128a15f88-kube-api-access-h9zpb\") pod \"maas-api-revoke-sub-89c66b3f-b658t\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:46.072051 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:46.071968 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:46.446964 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:46.446937 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t"] Oct 31 16:38:46.447151 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:38:46.447125 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87abeeb0_1509_49af_8c1d_1fd128a15f88.slice/crio-72fe54bc5e91d1faf4caeee8f8c02c1272ee605a1972ed8393b3d61e71ab5e26 WatchSource:0}: Error finding container 72fe54bc5e91d1faf4caeee8f8c02c1272ee605a1972ed8393b3d61e71ab5e26: Status 404 returned error can't find the container with id 72fe54bc5e91d1faf4caeee8f8c02c1272ee605a1972ed8393b3d61e71ab5e26 Oct 31 16:38:46.650413 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:46.650379 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" event={"ID":"87abeeb0-1509-49af-8c1d-1fd128a15f88","Type":"ContainerStarted","Data":"72fe54bc5e91d1faf4caeee8f8c02c1272ee605a1972ed8393b3d61e71ab5e26"} Oct 31 16:38:47.320223 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:38:47.320190 4399 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87abeeb0_1509_49af_8c1d_1fd128a15f88.slice/crio-c7495f4df3763ccea1e70ca41376e7ade717dd33d1e2ea5901df5d982a7d3cf8.scope\": RecentStats: unable to find data in memory cache]" Oct 31 16:38:47.320664 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:38:47.320317 4399 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87abeeb0_1509_49af_8c1d_1fd128a15f88.slice/crio-c7495f4df3763ccea1e70ca41376e7ade717dd33d1e2ea5901df5d982a7d3cf8.scope\": RecentStats: unable to find data in memory cache]" Oct 31 16:38:47.658051 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:47.657966 4399 generic.go:356] "Generic (PLEG): container finished" podID="87abeeb0-1509-49af-8c1d-1fd128a15f88" containerID="c7495f4df3763ccea1e70ca41376e7ade717dd33d1e2ea5901df5d982a7d3cf8" exitCode=0 Oct 31 16:38:47.658051 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:47.658040 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" event={"ID":"87abeeb0-1509-49af-8c1d-1fd128a15f88","Type":"ContainerDied","Data":"c7495f4df3763ccea1e70ca41376e7ade717dd33d1e2ea5901df5d982a7d3cf8"} Oct 31 16:38:49.114718 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.111889 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:49.203725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.203029 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-openshift-ca\") pod \"87abeeb0-1509-49af-8c1d-1fd128a15f88\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " Oct 31 16:38:49.203725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.203115 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87abeeb0-1509-49af-8c1d-1fd128a15f88-kube-api-access-h9zpb\" (UniqueName: \"kubernetes.io/projected/87abeeb0-1509-49af-8c1d-1fd128a15f88-kube-api-access-h9zpb\") pod \"87abeeb0-1509-49af-8c1d-1fd128a15f88\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " Oct 31 16:38:49.203725 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.203229 4399 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-xks-ca\") pod \"87abeeb0-1509-49af-8c1d-1fd128a15f88\" (UID: \"87abeeb0-1509-49af-8c1d-1fd128a15f88\") " Oct 31 16:38:49.208720 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.207890 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-openshift-ca" pod "87abeeb0-1509-49af-8c1d-1fd128a15f88" (UID: "87abeeb0-1509-49af-8c1d-1fd128a15f88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 16:38:49.215502 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.215473 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87abeeb0-1509-49af-8c1d-1fd128a15f88-kube-api-access-h9zpb" pod "87abeeb0-1509-49af-8c1d-1fd128a15f88" (UID: "87abeeb0-1509-49af-8c1d-1fd128a15f88"). InnerVolumeSpecName "kube-api-access-h9zpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 16:38:49.216083 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.216058 4399 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-xks-ca" pod "87abeeb0-1509-49af-8c1d-1fd128a15f88" (UID: "87abeeb0-1509-49af-8c1d-1fd128a15f88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 16:38:49.308781 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.306388 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-openshift-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:49.308781 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.306429 4399 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9zpb\" (UniqueName: \"kubernetes.io/projected/87abeeb0-1509-49af-8c1d-1fd128a15f88-kube-api-access-h9zpb\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:49.308781 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.306446 4399 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87abeeb0-1509-49af-8c1d-1fd128a15f88-maas-api-xks-ca\") on node \"ip-10-0-137-160.us-west-1.compute.internal\" DevicePath \"\"" Oct 31 16:38:49.713727 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.710482 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" event={"ID":"87abeeb0-1509-49af-8c1d-1fd128a15f88","Type":"ContainerDied","Data":"72fe54bc5e91d1faf4caeee8f8c02c1272ee605a1972ed8393b3d61e71ab5e26"} Oct 31 16:38:49.713727 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.710552 4399 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72fe54bc5e91d1faf4caeee8f8c02c1272ee605a1972ed8393b3d61e71ab5e26" Oct 31 16:38:49.713727 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.710628 4399 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c66b3f-b658t" Oct 31 16:38:49.947268 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.946798 4399 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-hvzpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 16:38:49.947268 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:38:49.946862 4399 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-hvzpj" podUID="f96f57e6-3024-4ef2-b7fb-8a49086ce52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 16:38:56.206414 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:38:56.206372 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:39:09.048109 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:09.048021 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q"] Oct 31 16:39:09.097743 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:09.097691 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c086957d-4424q"] Oct 31 16:39:10.896071 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:10.896036 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c42de4-2a0e-47e2-8c97-3473676e6e69" path="/var/lib/kubelet/pods/08c42de4-2a0e-47e2-8c97-3473676e6e69/volumes" Oct 31 16:39:15.565879 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:15.565834 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b9bfdbc7b-7jf7k_1404f6c8-54fc-49bc-b514-3e6956e20350/authorino/0.log" Oct 31 16:39:26.206582 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:39:26.206536 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:39:27.262174 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:27.262145 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-78f488d89-cv2qm_bd10b679-de31-4629-8256-cf75f52e76c4/manager/1.log" Oct 31 16:39:27.793124 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:27.793095 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p2f7_e191e1bb-a406-4fa3-b9bf-b27502111962/manager/0.log" Oct 31 16:39:29.307534 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:29.307508 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knls7_698a358f-ef89-4279-a36c-dd211218127e/server/0.log" Oct 31 16:39:29.846618 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:29.846584 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-28xqm_f35f6cb1-6438-4d87-a0ac-078e5960373e/proxy/0.log" Oct 31 16:39:30.513217 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:30.513186 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-78f488d89-cv2qm_bd10b679-de31-4629-8256-cf75f52e76c4/manager/0.log" Oct 31 16:39:31.774460 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:31.774425 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0c8c7538-4pc9p_cddb7bbc-ab64-479d-87db-93837af0bff7/revoke-keys/0.log" Oct 31 16:39:32.013381 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:32.013340 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-187a860f-ph2ms_48671135-10fb-48a2-b6a6-ca276fdaa76d/revoke-keys/0.log" Oct 31 16:39:32.236112 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:32.236085 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-1a0d4800-2442r_89613097-30b1-4a69-86f8-55f7312f5c5d/revoke-keys/0.log" Oct 31 16:39:32.450742 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:32.450710 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-502a5a0a-v8862_6559c121-9218-4b26-a1c3-684bb352e257/revoke-keys/0.log" Oct 31 16:39:32.674336 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:32.674266 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-650796a4-lz98v_f1401eb6-f43f-45f5-a1be-dddfb77a4c3a/revoke-keys/0.log" Oct 31 16:39:32.895186 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:32.895157 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-655f22d7-c2d9q_58a78b86-975c-4330-a195-e73bccef03f2/revoke-keys/0.log" Oct 31 16:39:33.118425 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:33.118393 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7426273c-hlzcj_b8237144-e60c-4b26-9122-daa6afa9939b/revoke-keys/0.log" Oct 31 16:39:33.342738 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:33.342671 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-89c66b3f-b658t_87abeeb0-1509-49af-8c1d-1fd128a15f88/revoke-keys/0.log" Oct 31 16:39:33.563313 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:33.563285 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9d86b2c1-d2skt_724e729b-d584-4654-b406-2a2f0f23948d/revoke-keys/0.log" Oct 31 16:39:33.786953 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:33.786923 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-aaa8fc1d-d775b_9f46de05-d0b4-415b-bc5d-fafc2ea30319/revoke-keys/0.log" Oct 31 16:39:33.994507 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:33.994476 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ae20d120-f4q4g_52d73349-d45f-4237-b0d5-a59632e1e69d/revoke-keys/0.log" Oct 31 16:39:34.218375 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:34.218347 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cfebeab1-7svjn_c7fb18cb-3a49-4d2a-a032-6a433eb3764f/revoke-keys/0.log" Oct 31 16:39:34.437637 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:34.437559 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d1431612-dxjx9_82157379-54a9-461f-aa1a-f6e443df2d2d/revoke-keys/0.log" Oct 31 16:39:34.654598 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:34.654574 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2f6cb25-zx4vk_7bb82c93-ba0a-4b08-9336-8b2f8fccd2db/revoke-keys/0.log" Oct 31 16:39:34.860970 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:34.860939 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e62685e0-lt4q9_39528537-4130-47e3-88e7-4b5c6bdf6e66/revoke-keys/0.log" Oct 31 16:39:35.078335 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:35.078309 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f1efce29-wkrnn_bed9cc4f-cffb-4774-b37b-cf729e8ebaa6/revoke-keys/0.log" Oct 31 16:39:35.293891 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:35.293860 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-7259x_8100f287-aef3-49cf-9fb4-a6363db3e4b1/postgres/0.log" Oct 31 16:39:37.234301 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:37.234276 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b9bfdbc7b-7jf7k_1404f6c8-54fc-49bc-b514-3e6956e20350/authorino/0.log" Oct 31 16:39:39.166140 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:39.166102 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-s7mjs_338dae66-850f-4bbc-ae9e-da170cf58631/manager/0.log" Oct 31 16:39:40.807684 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:40.807637 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b2w98_c9a93516-7ed3-4254-ab04-59dbe7552378/discovery/0.log" Oct 31 16:39:41.344167 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:41.344127 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9d7844885-hs7vq_69e742a6-7f45-426c-8b68-c8819d5d294e/kube-auth-proxy/0.log" Oct 31 16:39:42.061422 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:42.061394 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-hvzpj_f96f57e6-3024-4ef2-b7fb-8a49086ce52d/istio-proxy/0.log" Oct 31 16:39:43.661195 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:43.661155 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-vfnzr_c09561dd-b9cb-49e2-841d-54d528822afb/istio-proxy/0.log" Oct 31 16:39:43.880185 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:43.880140 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-xgc5z_116d08e1-3b25-43fb-92e9-2b960cd2e50d/istio-proxy/0.log" Oct 31 16:39:46.119353 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:46.119317 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-v7plc_c55db056-9324-4750-b1fb-49c4b40c3e92/main/0.log" Oct 31 16:39:46.615478 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:46.615445 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-rm55g_5d6f4e52-0d41-4213-941c-2b63c79cb2db/main/0.log" Oct 31 16:39:47.065957 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:47.065934 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8kfk98_dea35ba4-3d75-485a-9fd0-c91e74f8f62b/main/0.log" Oct 31 16:39:47.521362 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:47.521332 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-bggpb_33d1c606-bde8-4f26-9ebb-c51a6a7ab576/main/0.log" Oct 31 16:39:52.045232 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:52.045171 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r"] Oct 31 16:39:52.094545 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:52.094517 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1a0d4800-2442r"] Oct 31 16:39:52.895832 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:52.895799 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89613097-30b1-4a69-86f8-55f7312f5c5d" path="/var/lib/kubelet/pods/89613097-30b1-4a69-86f8-55f7312f5c5d/volumes" Oct 31 16:39:56.206202 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:39:56.206163 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:39:58.832565 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:58.832531 4399 scope.go:122] "RemoveContainer" containerID="58e7b951db52a25a1717cb06c42f07bfffb0acc5fb09e7311fa1f0eadeba14d4" Oct 31 16:39:58.842691 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:39:58.842670 4399 scope.go:122] "RemoveContainer" containerID="9a13f6b8ad4f1708fea99b76c5c855fe83b5a24ed72daa4dbf489dd42b68f657" Oct 31 16:40:03.045301 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:03.045265 4399 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn"] Oct 31 16:40:03.094122 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:03.094093 4399 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f1efce29-wkrnn"] Oct 31 16:40:04.895221 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:04.895190 4399 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bed9cc4f-cffb-4774-b37b-cf729e8ebaa6" path="/var/lib/kubelet/pods/bed9cc4f-cffb-4774-b37b-cf729e8ebaa6/volumes" Oct 31 16:40:08.125344 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:08.125310 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fjcpl_8c8ff594-7d79-4f56-9eb4-a7af14808c03/global-pull-secret-syncer/0.log" Oct 31 16:40:11.525850 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:11.525807 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-78f488d89-cv2qm_bd10b679-de31-4629-8256-cf75f52e76c4/manager/1.log" Oct 31 16:40:11.542258 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:11.542236 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-78f488d89-cv2qm_bd10b679-de31-4629-8256-cf75f52e76c4/manager/0.log" Oct 31 16:40:12.010929 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:12.010897 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p2f7_e191e1bb-a406-4fa3-b9bf-b27502111962/manager/0.log" Oct 31 16:40:12.198776 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:12.198727 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tqg7k_0eeb3c8b-578d-4a72-af8f-9c8c53887516/konnectivity-agent/0.log" Oct 31 16:40:13.008297 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:13.008270 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knls7_698a358f-ef89-4279-a36c-dd211218127e/server/0.log" Oct 31 16:40:13.020501 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:13.020475 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-160.us-west-1.compute.internal_dc41cca5aaa38cfa8c691e2d5baf2778/haproxy/0.log" Oct 31 16:40:14.817935 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:14.817902 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-28xqm_f35f6cb1-6438-4d87-a0ac-078e5960373e/proxy/0.log" Oct 31 16:40:18.103933 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:18.103880 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b9bfdbc7b-7jf7k_1404f6c8-54fc-49bc-b514-3e6956e20350/authorino/0.log" Oct 31 16:40:18.229592 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:18.229568 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-78f488d89-cv2qm_bd10b679-de31-4629-8256-cf75f52e76c4/manager/1.log" Oct 31 16:40:18.246562 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:18.246540 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-78f488d89-cv2qm_bd10b679-de31-4629-8256-cf75f52e76c4/manager/0.log" Oct 31 16:40:18.714815 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:18.714790 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p2f7_e191e1bb-a406-4fa3-b9bf-b27502111962/manager/0.log" Oct 31 16:40:19.375861 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.375801 4399 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb"] Oct 31 16:40:19.376570 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.376554 4399 cpu_manager.go:414] "containerMap: removing container" podUID="87abeeb0-1509-49af-8c1d-1fd128a15f88" containerName="revoke-keys" Oct 31 16:40:19.376570 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.376571 4399 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87abeeb0-1509-49af-8c1d-1fd128a15f88" containerName="revoke-keys" Oct 31 16:40:19.376728 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.376716 4399 memory_manager.go:365] "RemoveStaleState removing state" podUID="87abeeb0-1509-49af-8c1d-1fd128a15f88" containerName="revoke-keys" Oct 31 16:40:19.382765 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.382687 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.414765 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.414737 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb"] Oct 31 16:40:19.423465 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.423429 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-lib-modules\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.423603 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.423478 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-proc\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.423603 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.423503 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-sys\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.423603 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.423573 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nszmk\" (UniqueName: \"kubernetes.io/projected/facd167f-a1d6-455c-84fb-05aa3442a97d-kube-api-access-nszmk\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.423841 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.423659 4399 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-podres\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.434575 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.434531 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9r2f\"/\"openshift-service-ca.crt\"" Oct 31 16:40:19.434575 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.434549 4399 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b9r2f\"/\"kube-root-ca.crt\"" Oct 31 16:40:19.434742 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.434580 4399 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b9r2f\"/\"default-dockercfg-ztmvf\"" Oct 31 16:40:19.524395 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.524354 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nszmk\" (UniqueName: \"kubernetes.io/projected/facd167f-a1d6-455c-84fb-05aa3442a97d-kube-api-access-nszmk\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.524749 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.524481 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-podres\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.524749 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.524535 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-lib-modules\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.524749 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.524571 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-proc\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.524749 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.524604 4399 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-sys\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.524749 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.524615 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-podres\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.524749 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.524680 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-proc\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.524749 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.524672 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-sys\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.525462 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.524800 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/facd167f-a1d6-455c-84fb-05aa3442a97d-lib-modules\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.581089 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.581063 4399 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nszmk\" (UniqueName: \"kubernetes.io/projected/facd167f-a1d6-455c-84fb-05aa3442a97d-kube-api-access-nszmk\") pod \"perf-node-gather-daemonset-4zwzb\" (UID: \"facd167f-a1d6-455c-84fb-05aa3442a97d\") " pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.698307 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.698285 4399 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:19.988280 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:19.988202 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knls7_698a358f-ef89-4279-a36c-dd211218127e/server/0.log" Oct 31 16:40:20.070937 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:20.070909 4399 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb"] Oct 31 16:40:20.072288 ip-10-0-137-160 kubenswrapper[4399]: W1031 16:40:20.072255 4399 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfacd167f_a1d6_455c_84fb_05aa3442a97d.slice/crio-f23e60ae9c565aa90eb7b1983c4f7df4de4ebfa90605b1b3ffc94ddc3b40053c WatchSource:0}: Error finding container f23e60ae9c565aa90eb7b1983c4f7df4de4ebfa90605b1b3ffc94ddc3b40053c: Status 404 returned error can't find the container with id f23e60ae9c565aa90eb7b1983c4f7df4de4ebfa90605b1b3ffc94ddc3b40053c Oct 31 16:40:20.250675 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:20.250573 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" event={"ID":"facd167f-a1d6-455c-84fb-05aa3442a97d","Type":"ContainerStarted","Data":"ac70c23b0f645a9dd44f869862539f588d513c1613571bfcb5d88d11e975840c"} Oct 31 16:40:20.250865 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:20.250736 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:20.250865 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:20.250773 4399 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" event={"ID":"facd167f-a1d6-455c-84fb-05aa3442a97d","Type":"ContainerStarted","Data":"f23e60ae9c565aa90eb7b1983c4f7df4de4ebfa90605b1b3ffc94ddc3b40053c"} Oct 31 16:40:20.363206 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:20.363153 4399 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" podStartSLOduration=1.363137315 podStartE2EDuration="1.363137315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 16:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 16:40:20.361398993 +0000 UTC m=+6863.572640406" watchObservedRunningTime="2026-10-31 16:40:20.363137315 +0000 UTC m=+6863.574378726" Oct 31 16:40:21.696789 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:21.696754 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-28xqm_f35f6cb1-6438-4d87-a0ac-078e5960373e/proxy/0.log" Oct 31 16:40:21.863500 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:21.863470 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-s7mjs_338dae66-850f-4bbc-ae9e-da170cf58631/manager/0.log" Oct 31 16:40:24.666661 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:24.666627 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-78f488d89-cv2qm_bd10b679-de31-4629-8256-cf75f52e76c4/manager/1.log" Oct 31 16:40:24.682319 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:24.682292 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-78f488d89-cv2qm_bd10b679-de31-4629-8256-cf75f52e76c4/manager/0.log" Oct 31 16:40:25.134590 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:25.134561 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-2p2f7_e191e1bb-a406-4fa3-b9bf-b27502111962/manager/0.log" Oct 31 16:40:26.205744 ip-10-0-137-160 kubenswrapper[4399]: E1031 16:40:26.205691 4399 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zlmzj" containerName="extractor" Oct 31 16:40:26.266758 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:26.266735 4399 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b9r2f/perf-node-gather-daemonset-4zwzb" Oct 31 16:40:26.293366 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:26.293347 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-knls7_698a358f-ef89-4279-a36c-dd211218127e/server/0.log" Oct 31 16:40:28.088511 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:28.088476 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-28xqm_f35f6cb1-6438-4d87-a0ac-078e5960373e/proxy/0.log" Oct 31 16:40:29.639283 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:29.639253 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-gxcct_8a90dc3f-448b-4dd1-a571-6a9b1dc17349/monitoring-plugin/0.log" Oct 31 16:40:30.062291 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:30.062256 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2jxj6_f6e52f66-5e0e-4cbb-852d-fd4aa259d271/node-exporter/0.log" Oct 31 16:40:30.479559 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:30.479535 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2jxj6_f6e52f66-5e0e-4cbb-852d-fd4aa259d271/kube-rbac-proxy/0.log" Oct 31 16:40:30.896439 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:30.896355 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2jxj6_f6e52f66-5e0e-4cbb-852d-fd4aa259d271/init-textfile/0.log" Oct 31 16:40:31.705821 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:31.705788 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-b2w98_c9a93516-7ed3-4254-ab04-59dbe7552378/discovery/0.log" Oct 31 16:40:32.125589 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:32.125524 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-9d7844885-hs7vq_69e742a6-7f45-426c-8b68-c8819d5d294e/kube-auth-proxy/0.log" Oct 31 16:40:32.767793 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:32.767766 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-hvzpj_f96f57e6-3024-4ef2-b7fb-8a49086ce52d/istio-proxy/0.log" Oct 31 16:40:33.825119 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:33.825087 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-gz5c7_166f6dae-eda8-4568-8183-20db7b0dd0b6/kube-rbac-proxy-main/0.log" Oct 31 16:40:33.957425 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:33.957393 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-vfnzr_c09561dd-b9cb-49e2-841d-54d528822afb/istio-proxy/0.log" Oct 31 16:40:34.183638 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:34.183554 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-xgc5z_116d08e1-3b25-43fb-92e9-2b960cd2e50d/istio-proxy/0.log" Oct 31 16:40:34.272031 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:34.272003 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-gz5c7_166f6dae-eda8-4568-8183-20db7b0dd0b6/kube-rbac-proxy-self/0.log" Oct 31 16:40:34.690474 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:34.690447 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-gz5c7_166f6dae-eda8-4568-8183-20db7b0dd0b6/openshift-state-metrics/0.log" Oct 31 16:40:35.117895 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:35.117847 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30e0121f-40e7-4a32-b0a7-e635216391a2/prometheus/0.log" Oct 31 16:40:35.532307 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:35.532280 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30e0121f-40e7-4a32-b0a7-e635216391a2/config-reloader/0.log" Oct 31 16:40:35.949287 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:35.949258 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30e0121f-40e7-4a32-b0a7-e635216391a2/thanos-sidecar/0.log" Oct 31 16:40:36.364791 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:36.364691 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30e0121f-40e7-4a32-b0a7-e635216391a2/kube-rbac-proxy-web/0.log" Oct 31 16:40:36.781006 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:36.780977 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30e0121f-40e7-4a32-b0a7-e635216391a2/kube-rbac-proxy/0.log" Oct 31 16:40:37.197574 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:37.197549 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30e0121f-40e7-4a32-b0a7-e635216391a2/kube-rbac-proxy-thanos/0.log" Oct 31 16:40:37.614051 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:37.613954 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_30e0121f-40e7-4a32-b0a7-e635216391a2/init-config-reloader/0.log" Oct 31 16:40:39.339976 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:39.339951 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7585f46c4d-l65j7_3a7f6a15-9317-42d1-8783-95ea990102a2/telemeter-client/0.log" Oct 31 16:40:39.756525 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:39.756489 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7585f46c4d-l65j7_3a7f6a15-9317-42d1-8783-95ea990102a2/reload/0.log" Oct 31 16:40:40.172340 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:40.172274 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7585f46c4d-l65j7_3a7f6a15-9317-42d1-8783-95ea990102a2/kube-rbac-proxy/0.log" Oct 31 16:40:40.595453 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:40.595416 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6646454b57-j46xn_8008214d-c86a-4ff0-9838-54c39f5d2a81/thanos-query/0.log" Oct 31 16:40:40.736302 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:40.736270 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b9bfdbc7b-7jf7k_1404f6c8-54fc-49bc-b514-3e6956e20350/authorino/0.log" Oct 31 16:40:41.016115 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:41.016085 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6646454b57-j46xn_8008214d-c86a-4ff0-9838-54c39f5d2a81/kube-rbac-proxy-web/0.log" Oct 31 16:40:41.431722 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:41.431622 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6646454b57-j46xn_8008214d-c86a-4ff0-9838-54c39f5d2a81/kube-rbac-proxy/0.log" Oct 31 16:40:41.849070 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:41.849041 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6646454b57-j46xn_8008214d-c86a-4ff0-9838-54c39f5d2a81/prom-label-proxy/0.log" Oct 31 16:40:42.093051 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:42.093021 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-s7mjs_338dae66-850f-4bbc-ae9e-da170cf58631/manager/0.log" Oct 31 16:40:42.266615 ip-10-0-137-160 kubenswrapper[4399]: I1031 16:40:42.266586 4399 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6646454b57-j46xn_8008214d-c86a-4ff0-9838-54c39f5d2a81/kube-rbac-proxy-rules/0.log"